One of our all time most successful tests was related to this story and happened almost by accident.
The client had just rolled out new packaging and wanted help validating the change statistically, very similar to this situation.
Their metric was measuring upsells to a premium product after a purchase, and like Candy Japan, found statistically insignificant results.
But the new packaging was beautiful and we wanted to test it on their landing page.
We tried just replacing the old product image on landing page. Meh results, no real bump.
We tried increasing the size of the image + circling with "Improved" (no joke). 10% bump. Interesting.
We hired a professional model to hold the product and smile. 15% bump. !?
We hired a group of professional models to hold the product and smile. 40% bump in sales. !@%K$!?
We re-hired the same group of people to hold product, look SUPER excited (jumping up and down like their team just won the superbowl). 50%+ bump in sales.
No joke.
If Candy Japan is reading and you'd like some help recreating I would not be shocked to see something similarly group oriented, visceral, and beautiful bump yeh.
>10% bump.
>15% bump.
>40% bump in sales.
>50%+ bump in sales.
Was that cumulative (i.e., 1.1 * 1.15 * 1.4 * 1.5 ≈ 2.7 total) or relative to the base figure?
How many sales would Amazon have today if they went back to their 2000s website look?
The decorated box might not even have the best drawing, but indicates that the salespeople went the extra mile
In my mind's eye, that'd remain classy enough not to be creepy/cringy/embarrassing (as https://news.ycombinator.com/item?id=11641715 complains), but would also have sort of a personal touch to it.
Having a couple rubber stamps made and stamping each box during packaging wouldn't probably cost an awful lot...
After all, his main edge over the local japanese stores is authenticity. The fact that it comes "directly" from Japan.
In other words, the old box, while plain, adds a certain intimacy. It doesn't look like a huge multinational candy company sent it. The new box, though, looks basically like any other piece of Japanese media.
The Japanese illustrated box is innocent and kind of cute (certainly not "rapey" as another commenter put it) and in a Western country, stands out from the things around it. Perhaps if you're in Japan, surrounded by Japanese media it might seem generic but I don't think that's a problem elsewhere.
* It looks childish. If you are familiar with the Japanese style of advertising etc., probably not an issue, but as a box to be sent anywhere and to be received in a variety of settings (workplaces etc.) it seems like a poor choice.
* Anime may have poor connotations (unfair IMO) as pointed out elsewhere in the thread, which will reflect on the receiver.
* It feels corporate / like advertising. It may seem silly, but I'm more likely to continue subscribing to a small company if I'm on the fence about canceling.
* It feels like a waste of money. The current packaging is utilitarian, which signals that money is being spent only where it needs to be. The new packaging feels like an unnecessary indulgence, which would make me think I'm getting less value out of the subscription.
was the old box that blue plain box? It's not intimate to me. Compared to the new one, it looks kind of homely
- The anime picture is embarrassing to receive in the mail / at work.
- The picture itself is kind of creepy, someone walking past my desk once even described it as 'rapey'.
- I can reuse the box.
- I am not 6.
This person may have some deeper issues than just the innocent picture on this box.
Someone who admires Japanese culture will likely not have the same reaction as someone with little exposure to Japanese culture.
The picture is not even on the same planet as 'rapey' Whoever that was either said it to get a rise or they have some distinct issues.
One solution, perhaps, is avail both packaging options to buyers.
The typeface and font, I think, could be improved. It kind of looks like an afterthought. I don't see "branding". It's not Lobster or Comic Sans... but...
One problem is that even if the person receiving the box is exposed to Japanese culture, their coworkers probably aren't.
I think you're projecting your own issues quite considerably.
Maybe I should start calling this the plain vs. rapey test.
Have you tried to run introduction offers? 50% off for first shipment for example? If I were you I would try to put the countdown for the next shipment on top of your site. Thats a really powerful psychological factor.
Your CTA is also a little unclear, why do I need to first get an email, then a link? The buy process should be straight forward, not through my email. And if you have a really good reason for doing it this way, you should atleast tell me why I need to leave my email.
I really like the cartoon that explains the process. Very good.
Regarding packaging design, we did a huge remake in 2013. We increased our prices with 30%, and redesigned packaging completely. Doubled our LTV, insane gains. Just keep testing, but try to change provider for carton. We pay $0,1 USD for bigger cartons then yours, in one of the most expensive countries in the world. (With plastic window for address field) We chose to put our three USP on the back of the shipment box, to remind your customers every time they receive a shipment why they subscribed in the first place.
Keep testing!
* What does LTV mean?
* What does USP mean?
USP = Unique Selling Point = Why your subscription gives the subscriber something they can't get anywhere else
Why not poll your subscribers or (assuming you can get the costs to work out) have an option for anime art. Personally I wouldn't want the anime style box, something custom and simpler would be fine.. but thats my personal opinion.
I suddenly get p-values now. Why has no one ever explained it like this before?
Which package design would you rather receive?
The white one with illustration 79%
The plain dark one 21%
Total votes: 52
The white box would be better advertising for you in my case :)
If I went to a friends house and saw this, I would never remember the name. If you could combine "making the purchaser look cool" with timing-relevancy, I think your packaging improvement would pay dividends.
Put another way, I believe your A/B ratio test is actually X*Y/Z where X/Z and Y/Z are both < 1. Synergy.
Terrible idea, (possibly) useful for analogy: Instead of optimizing the unboxing experience alone, if you also included cards with QR codes, where every QR code immediately triggers a free candy bar (with user data entry on a mobile app). It should also provide social value to the host in realtime; maybe they also receive an immediate notification that their guest scanned their card plus some other bonus I can't think of.
The point is to get closer to the conversion/churn decision, or provide more value to your consumer.
I used to receive Candy Japan in the "plain" box to Australia, and the customs sticker declaring the value & contents took up most of the space on the box.
https://www.quora.com/What-statistical-test-should-be-used-f...
(Of course, even better would be a Bayesian decision analysis which takes into account the full posterior distribution, costs, and benefits, allowing one to decide whether the evidence is enough to justify the more expensive box or whether one should extend the experiment further to collect more data.)
* quick, if conversions are 0.25%, what does a 0.1SD mean difference between arms of an A/B test mean? 'Uh...' Exactly.
All of this is "I think" and "IMO" of course.
