Walling's Stair Step Method - Shopify App Deployment (Part 4)

Introduction

This post has taken me longer than the previous ones. Actually getting things out is harder on a different scale than just making something work. The listing process and deployment is more complex than it initially looks like, and everything has to be precisely right for it to work at all. There is little room for half-baked solutions here.

Initial Deployment And Listing

After Basic MVP

I’ve gotten to the point where the initial implementation is done. Now - is it deployment and putting it up on the marketplace? Questions that arise for me are whether Shopify supports hosting the Remix apps themselves without hassle or I have to host it on my own rig, how is user management done and how much do I have to keep track of them, should I write custom limitation code for the ChatGPT API not to be hit too frequently.

Remember - the point here is to get something out as quickly as possible, so I experience the full process end to end, so it becomes a known path for me, something in my brain that feels like I can easily do it again and at least 3x as fast.

I think the twenty dollars I spent on ChatGPT right now isn’t a big deal for me, people can use it if they find it useful, and spend it if they want. So I won’t be limiting the app usage on my first deploy up. I think I’ll put some limits on it as soon as I get some real traction! That twenty dollars spent is a blessing that shows people use it!

Okay, so let’s just try to deploy it and see where we get stuck. Based on the Shopify docs on deployment it seems like you can deploy it however you want, but I am not completely sure what that means. So let’s go with the easiest option - let Shopify do it for me. Interesting it says that the different components of the app, including the extension I just wrote, are deployed separately. Seems like you can create both non-production and production deployments by creating another Shopify app with the same codebase, but right now on this scale I want to keep this as simple as possible, so I’ll go straight to production with the codebase I already have.

Even though the documentation initially said that you can also deploy the apps from the Partner Dashboard, then deeper in the documentation it does not seem to support that path actually and suggests two other hosting providers. Hmm, that is confusing. So I go to the Partners Dashboard, which is like a separate Shopify subdomain where it’s a different interface and options of managing your relationship with Shopify as more of a business rather than a store. There in the Apps tab I actually see that there seems to be an option to both publish and distribute this app immediately to the public. I clicked on the button and proceeded, and I get this page where there’s another button to create a listing for the app. I am not sure if the app is deployed into production already or not, but let’s hope so (future me: this was not the case).

So the distribution page with the button also has a criteria checklist to get some nifty badges on the app in the listing. Which is definitely needed as trying to give a decently good shot at the first marketing side of things is important if I expect to get even a tiny piece of information or feedback from the users. There seems to also be automatic checks that Shopify does, which I am now waiting for. It seems like most of the criteria expected seem alright to me, but now I am worried this whole situation means that the encapsulating app and the extension are listed on the app store separately (future me: they’re not). But for now, seems like the app will be reviewed after creating the initial listing for it.

Seems like the app itself is somewhat live, and when I go to the extensions section, then it seems to be correctly welded into that app. Hmm, let’s hope so! There’s also a versions section that seems to be empty. I can create a release, but I wonder if that’s any different from what’s already out there. In the API access tab I see one of the configured scopes is write_products, but that’s definitely not necessary. All the app is doing is reading and giving suggestions. So in order to make the whole installation process more oiled for the first potential customers, I definitely need to get this fixed in the configuration file first (future me: I actually went back to using the full write_products because Shopify doesn’t seem to separate access gating very well yet).

Going back to the criteria page, seems like one has to apply to get the badges. I wonder if I can do this before the first app store launch. I should find this out (future me: I didn’t).

App Store Listing

Now that I am trying to put the app up on the app store, I see that they want a one-time 99 dollar listing fee. Which is technically fine if I never have to pay this again for any of my apps, but that is bad news if it does not work like that (future me: it’s account based). It seems to be a registration for the 0% revenue sharing plan up to 1M, but given that at this point I just want to test the market, I don’t mind the 15% revenue share if I can avoid paying the 99 dollars (future me: I couldn’t). At least it seems like the fee is per partner account, not per app.

I think this small barrier is good in a way, that keeps off less serious people off the platform. But that also means that building one app and spreading it on multiple platforms is probably a worse idea than to focus on one platform and keep building on that, at least for the time being, because of the opportunity costs other platforms might also have.

I bit the bullet and paid the fee, because there’s no good alternative at this point in time. I did pay under an individual right now, but it might happen that I need to switch it to a business later on if I don’t want to be taxed more than necessary.

Now I see that the app is public (future me: that doesn’t mean almost anything as far as I can tell), but the listing is in draft state. Let’s see what is there to do… It seems like I need to build an app icon and do GDPR mandatory webhooks.

Starting with the icon, it needs to be 1200x1200. So I’ll open up Photoshop and cook something up quick. Given it’s about analyzing text sentiment magically, perhaps a simple crystal ball with some text would do. Actually, you know what, I found this nice picture of a stone that will work for me as it is super simple yet somewhat interesting. But next time seems like the logo needs to be very clear in very small formats too, and for this it wouldn’t work. So perhaps going with pixel motif would do good for me too on future apps.

GDPR Webhooks

With the mandatory GDPR webhooks, the good thing is that very little code has to actually be written. If the requests are rare enough, then you need to supply the necessary information or delete the customer data within 30 days. So that’s easier to do by hand for now, but I think I’d implement it after the first real request too (future me: it’s easier to just implement it for real).

Ah, now I see that the template that I built my app on actually had these webhooks already implemented as a skeleton, but throws a 404 instead. I’m now thinking what do I actually have to do after sending a 200 back. There’s two thoughts - either I can do the action immediately and return a 200, or I could save the information and do it manually later on. I still like the manual way more because then I am more aware of it what is happening, and I am not just randomly losing rows in my databases. All-in-all I still need to create a new table where to submit these requests into. Also, with deletion requests it’s a bit more self-evident, but with data requests I wonder if the request itself has information on where to send this information back to too. Ah yes, now that I check the documentation, the example data request actually has all this information.

I thought of doing a simple incrementing table with a plain JSON in its second column, but Copilot actually built me a really neat structure instead. Let’s hope it works (future me: it does).

model GdprRequest {
  id        Int      @id @default(autoincrement())
  shop      String
  email     String
  firstName String?
  lastName  String?
  phone     String?
  message   String?
  status    String?
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt
}

Actual Listing Preview

I’m glad you can preview the listing before actually submitting that. That means I can be more relaxed about it as long as I ensure it looks good to the customers.

There are countless different settings for the app submission page to change which I will go over now.

It was hard to find a (pre-defined selection of a) category for that listing, and it needs specific approval to submit it to multiple categories - which makes sense, because it would give out much more visibility for the app. But I went with Other Analytics as the category. It is closest to being an analytical tool rather than anything else.

Languages is an interesting thing here, but I feel like this has to only be English for now, because otherwise the prompt text should also be modified for the different languages when communicating with ChatGPT. But I assume that in the future it would be important to spread any app I create to as many languages as possible, e.g. Spanish, as there are so many people not working actively in English. But this here also made me realize that I do support only English product descriptions right now… which is funny, because it’s not even my mother tongue. I have become the culprit of over-anglicizing my life!

It is suggested you create a demo store showcasing the app’s capabilities. I think screenshots do it justice enough for now, but for other apps in the future, that seems like definitely a good way to promote the app. Same goes for the YouTube video section.

Legal-wise there has to be some sort of privacy policy somewhere too. I try to keep this as standard as possible. I make a route for that HTML on the app itself, so it’s automatically hosted with everything else. But once I did that, I have no idea what the absolute URL of this could be once someone starts using the app. Because the app exists relative to the customers admin panel. So it seems like it might need to actually exist somewhere weird instead for now. I’ll instead post it to Pastebin then.

Images

For the screenshots and images, I did do deliberate screenshots of the app in the previous post I can use, but I don’t have a marketing-esque image for a banner image right now. I’ll go with the same screenshot and see how the preview of the listing looks like. Okay, no, apparently they have to be 1600x900 exactly, and it is said most people browse the app store with their mobile phones, so it should be a mobile screenshot.

Even though I am somewhat annoyed by the fact that there is so much deliberate effort needed to just get the listing out, it does make sense, and I am happy I can polish it until it looks halfway decent. Plus, I shouldn’t forget that everything I do now is at least 3 times less time-consuming the next time.

As I went on to take screenshots of the app on my physical iPhone, I was happy to see it correctly tunneling my local environment, so I can use it from the official Shopify app, but unfortunately only the app part and not the extension. For the admin extension pointed to the product page, I don’t see the button for my extension, unfortunately. I do see the App action Run Flow automation, but not my own. So I wonder if I have left something undone or is that how it is testing-wise. And now that I am writing this out I realized that of course I can just test the mobile page with developer tools too.

For some weird reason looks like the mobile screenshots also have to be 1600x900 landscape, so I am padding them out for now. I don’t think Shopify wants screenshots in mobile landscape mode.

This part was surprisingly annoying, but it makes sense as pictures are the most attractive part of a listing usually. But I still don’t want to spend too much here because the goal here is to get the app out there first and then learn what’s really important. I feel like I will probably put more effort into this step on my next apps now that I have seen how it’s being shown.

Pricing

Under the pricing details now I see the reason why so many go with free trial offerings, it’s just one of the more reasonable defaults offered by Shopify. In order to get external billing support, you need to get explicit approval for that.

Now I don’t know whether I should ask any money for the app or no. It seems like there are dangers both ways. If it’s priced free, then I am basically paying for ChatGPT, but I get product fit signals. But then again, free product signals are weak signals, so I should set it at least 1 dollar. And because my costs are nonzero on the API, there is choice to either go monthly with the billing or based on API usage. But because I don’t have any metrics for the usage set up right now, I’ll just go with a small 10 dollar per month fee for now. Anyone with a half-decent store shouldn’t mind this amount if it helps at all. Plus I’d give people a free trial. At this point I should calculate on average how much a single product request would take money. When I check ChatGPT I don’t see a single cent taken for the requests I’ve done testing it, so I don’t really know. Okay, so basically I want a nonzero price for the app, but I don’t want to ask too much either because the utility of all this is pretty limited. Let’s go with 4.99 a month after a 3-day trial! Anything under that does not make a difference for anyone really if you’re running a business - either way it’s the price of an ice-cream. I do want to know whether it’s a mistake making it that cheap though. But we’ll see, and I’ll adapt.

Tracking

Very interesting to see that it supports listing usage out of the box. This is super helpful! I will definitely set the Google Analytics up for this.

Quality Changes

So now that I’ve completed all the required points in the listing, I do want to make sure that the first use is as smooth as possible by predicting a few simple failure modes. I know that this is extra work before anything gets out, but I don’t want it to be a 50% chance of things breaking either. I guess my OKR would be that out of 10 users that would install it on their stores, 9 people regularly don’t have things breaking for them. On any SaaS standard that is still a extremely high error rate, but here we emphasize speed.

The first thing I am thinking that is breaking the happy path is when any of the inputs are empty. That can be the vision text, that can be the product text. Before setting decent defaults, I will make it so that the user is noted if they need to do anything extra. Now that I’m thinking about it, it might be actually cool to set a placeholder text on the vision board that would take effect if nothing is written there. Something like optimizing for good style and legibility, with sprinkles of SEO. And make the empty product description also a happy path by saying that in the ChatGPT prompt to give general writing advice if none has been written. And I should put some character limits on these not to break things too hard. To make it more seamless for the user, I’ll just submit the first n characters and do an ellipsis after it.

I wanted to engineer the prompt in a manner that would give the user helpful suggestions if there are no descriptions, but the stability of the responses truly suffer with this. So instead of saying if-then statements within the prompt, I should do the separation in code myself first. I think I got it to a decent state. I did up the temperature when it helped writing the description based off the title that existed there. To my surprise even the GPT 3.5 is pretty good in creative writing, at least in this context. (future me: now that GPT4 turbo is out, it might be a good idea to use that instead, but until I see any traction I will not do that)

Next up I’d love to do a small transition state when the text hasn’t arrived yet, because I feel it would give a big impact on the experience.

After First Submission

After submitting, I was hit with three errors. One is that the GDPR webhooks didn’t work (future me: the URLs need to be written in a very specific way in the listing), the app name has to match, and lastly they need my phone number.

The GDPR webhooks not working made me think if what is actually being deployed to Shopify right now is the autogenerated application_url within the app config file in Cloudflare. I figured this out because as I was about to push the config up, it asked me whether I actually want to remove the auto-generated webhooks for the GDPR requests. And then it made me realize what it had done. It had built these URLs using the URI-s for these endpoints I had provided setting up the deployment before starting the listing (which were the automatically deployed Cloudflare ones). But that all means that the autogenerated Cloudflare endpoints are truly actually just for development work and not a configuration to be used in production. Which sorta makes sense, but also a bit counterintuitive. Like - Shopify can handle all this hosting already, so why would I need to do anything extra?

But now I remember, this Cloudflare URL is only a tunnel to my own local setup. It’s not actually deployed up anywhere. So even if I got to go through with this, that would mean the computer that I am developing this on acts as a server.

I do have a proper server of my own that can do all this, and I am happy to use it for this purpose. By the way, I think the extension part is actually hosted by Shopify themselves, it’s just the encapsulating app that I have to host myself (future me: the extension is deployed via npm run deploy). It now loops back to the previous problems I had getting the connections between these two pieces properly working, and now I understand that it is with purpose that the extension can only be a very thin, embedded frontend client.

Now I am finally properly reading the README.md file that has instructions and general ideas about deploying all this to production, but not a lot. A lot of it points to Remix documentation and Remix documentation itself points a lot to the idea of “hey you can do whatever you want” (future me: this is absolutely not true).

Deploying to Production (Environment Variables Headache)

First, let’s start by running NODE_ENV=production remix build and see what happens. This appears to hit me with a 'NODE_ENV' is not recognized as an internal or external command. Then I changed the npm build script to SET NODE_ENV=production & remix build because I am on Windows and that did the trick for this particular problem. Writing this I am hit with Unrecognized mode: production. But then I also realize that I probably don’t have remix CLI either. Which wasn’t necessary, as you can run remix commands through npx. I ran first NODE_ENV=production and on a second line ran npx remix build. Not sure why the proposed way in the package.json is so different. Perhaps that’s the way on Unix? That built the project as expected. Now I am not quite sure whether the code to deploy sits in the build folder or the public folder. Neither of these include anything that a normal static website would have like an index.html file. I did see an index.js though. (Future me: it’s the public folder for static assets, and build folder is for the backend)

Seems like it follows standard Node deployment rules and the build folder is what needs deploying. So next up I need to set up serving Node from my server.

As I am trying to figure out how to serve the app, to my surprise I cannot just deploy the built production files, but rather the standard way of deploying node apps is to clone all sources to the target machine, build it there, and run it like you would on local, just that with some extra environment variable that says it’s production. It’s one of these things nobody says explicitly, but everyone assumes. The question is now whether to set up a copying script for deployment, but I think I probably should actually use git for this as there are files that I wouldn’t want to copy over, like .env (future me: Remix thinks I cannot handle keeping this a secret in production, so it doesn’t allow using it).

After cloning and building the code on my VM, I get

Invalid appUrl provided. Please provide a valid URL

(Future me: This was the longest bane of my existence, and the solution is to force all environment variables down the npm start command’s throat)

Seems like before I can deploy the backend, I need to install the app with that configuration to Shopify first.

Even if I create a new toml config file, it does not do anything until it’s actually pushed to Shopify. Because logging in and managing via UI on the Linux VM is not possible, I figured the configuration file does not have to be on the machine at all. I can use it locally on my own computer and push it for Shopify to use. (future me: I still don’t quite know whether npm start on the production VM actually uses the toml files or not)

Seems like Shopify reads the dotenv (future me: it doesn’t) and its toml file to create environment variables to be used when it deploys things on the cloudflare domain for development, but these need to be explicitly set for the Debian environment if I want to run the app on this. This is why I get the invalid appUrl provided I think. Because these values are being read with the process env chain, I think I should be able to just change the .env file to provide all these values, really (future me: no, I can’t).

Once I got the actual prod app server running, it did not properly read the dotenv file on running. Well, I think it did for the shopify specific ones (future me: the Shopify environment variables aren’t read from .env), but not for the remix ones. It did not properly read the port it should run on. But then again that’s pretty much the only one here. I can just add it before I run npm. And that works as expected!

Apache Conf And Routing

Now, let’s check if I got my Apache conf correct. Seeing if a webpage is displayed on the expected url that I set for the frontend. But that failed and got a 404. I think it’s because only the assets are served on this URL without a deliberate index html file. I think it has to go through the remix backend to even be served anything. That makes sense given how the Remix file structures are. But I am afraid the static folder might break then. Even then, I am changing the Apache conf to account for this such that not just the api and auth endpoints, but also the root is being served by the remix backend. This would actually make sense! But I just need to somehow definitely still give permission to serve the user the frontend files.

As I enabled the root to be served by the backend, I got the right response, however as expected - without the frontend files, like css. So I am taking a look where it is searching for these. Even though the site that I created namely for this purpose was under path /nyx, it searches the /build folder on the domain root level, which is definitely wrong. I wonder if I can also tell Remix on build what the path prefix should be. I found it and this is called publicPath and can be written in remix.config.js. In order not to have clashes with my /nyx path and to simplify things, I’ll just use /nyx-static in the URL and try to use Alias for this in the Apache conf. Again and again I am surprised how intricate it all gets, just for getting out a simple app!

So this absolutely changed the request URLs to my joy. But seems like they’re not still being served. I think it might be the Alias for which I don’t have the mod enabled for. Unfortunately, it seems enabled already, which means the culprit might be somewhere else. I’ll check whether the permissions to these files are correct. But no, I did not type the path to the static files correctly! Okay and now I see that the folder that I am using was for the backend that is being run by remix server itself in build. I should have been using public/build instead.

Okay, now that this is working and the page seems to be loading correctly, it gets into an infinite redirect loop to itself for some reason. Now that I read off of Remix’s forums that running Remix on a subdirectory is not supported, heh. So I have to redo things. The reason why I wanted to avoid writing a new subdomain for every app I develop is that I would need to generate new certificates for them every time too. And that’s just extra work for every app. It seems like there’s a way to use a package to get the basePath addition, but since Remix is so heavily assuming the capability of running everything off of root for its router, I’ll avoid future problems and just run each app from its own subdomain. Unfortunately I’ve already named the first one shopify.silvertaza.com, assuming I’d gather together all apps under this name, but it’ll do for now then. My memory is pretty fresh about generating new domain certificates though, so wouldn’t hurt to do it one more time for nyx.silvertaza.com either.

Because there’s no good examples on the internet that I could find about an Apache conf that would actually work properly in this use case, I am adding a picture here. The certificates are generated using acme.

The Hard Stuff

After trying to fix the things from the first feedback where some of my routes got messed up, I am unable to use the app bridge anymore to do queries for some reason. Giving it some time to resolve itself. Until then, I see that on first install the loading of the main app page just takes too long, so I am putting a suspense block there as well. Probably need to create a new video as well showing that it might take time to load.

I try to simplify the routing, but I get Error: No Features were provided. So reverted back.

Even with the reverted code, I get permissions errors on making graphql requests to products, seems like you need write rights to do a reading query to graphql.

And again I get CORS problems. Had to rotate the keys, because they seemed to be permission errors.

Everything seems to go wrong…

So I am unable to make db requests from backend suddenly. Not sure if this is because of some Shopify update, or I have flunked something up. Need to figure this out.

Seems like the essential issue here is the invalid oauth callback. [shopify-app/ERROR] Error during OAuth callback | {error: Invalid OAuth callback.}

First way I am aiming to solve this is trying to go to an older commit that works. Checking out a commit from the past, pushing the config from that up, and then see if it works. It seems obvious, but it broke on the same commit, so I am not sure when it actually broke. At this point I have to assume it still is the code I have written, it’s just that when it still worked, I think it might have been using old pieces instead of what I thought was deployed. This lack of clarity isn’t exactly great for development, but I have to deal with it. Seems like there is some issue with browser cookies oauth_error=same_site_cookies, so I ensure that these are clean. Because of all my troubles started when I started adding production config, I feel this teaches me the importance of having separate apps for local development and production. Even though this is not how usually development is being done in modern web applications, then I might just need to adapt here. Do your development on one repo, and once you’re happy, you just merge the changes from the other origin to your production repo. But for now all I want is my local to work.

My other thought was to create a completely clean project and just be super additive with everything, and not delete or clean anything that Shopify had provided me first. Because I was not able to make it even work with older checkouts of code, I am going to start from zero and create separate dev and production environments for this app, even with the repository. I’ll name them nyx-dev and nyx-prod.

I have a hunch the whole problem might be that Shopify had somehow forgotten my secret key. And it was never written down in .env file, so I guess it had to be stored somewhere on the Shopify side (future me: this was not the case). Creating different repos will keep everything more clear with the configurations and I don’t need to do any switching between these. However not all work is lost, I still have the business logic correct, and the VM’s server setup does not need to change at all.

(future me: it is because Shopify updated their APIs without continuing support for the old ones, it brought me a lot of pain)

Next up

In the next post I will create clean repositories from scratch and successfully get it running again.