A single consumer-grade phone photo can make reliable skin-attribute readings fail. When we started Cutib in 2023, we expected the recommendation layer to be hardest: matching readings to ingredient compatibility, weighting product history, and surfacing suitable products for each profile. We were wrong. Turning an image into a dependable skin reading took our first eight months and most of our early iteration budget.
This post records what we built, where the first two versions failed, and which decisions made the output useful enough for real recommendations. The problem is not solved. But we learned what not to do first, and these lessons are worth recording while they remain fresh.
The input problem is harder than it looks
Consumer phone photos differ enormously. Lighting, skin tone, sensor quality, focal distance, angle, and mirror capture all change what the model sees. A pipeline trained mainly on controlled studio images fails in the field. When we started, most labeled datasets we found consisted of consistent studio shots from narrow demographic ranges, often under clinical lighting unlike the way people photograph their faces for product recommendations.
Our first version used a fine-tuned vision model trained mostly on curated datasets, with augmentation meant to mimic lighting changes. Lab results were good. In field tests, early-access users' photos degraded sharply under warm indoor incandescent light, bathroom fluorescents, and partial shadow. The model had learned associations valid only for its narrow training distribution, not the conditions it met in practice.
What the first two iterations missed
Version one made a structural error: skin-attribute prediction was treated as one multi-label classification task. One inference pass predicted every attribute, while common features consistently overwhelmed rarer losses during training, including sensitivity signals and specific dehydration patterns. Common characteristics were predicted well and rare ones poorly. For recommendations, that is the wrong tradeoff, since rare attributes often carry the most decision-relevant information.
Version two split the task. We separated coarse texture, specific condition detection, and undertone analysis into separate heads. Minority-class recall improved, but a new issue appeared: the heads did not share information about underlying image features. Each therefore worked harder than necessary and sometimes disagreed on edge cases.
The impasse broke when we treated the three attribute families as dependent inferences, not independent predictions. Barrier condition changes how surface oiliness should be read. Dehydration markers alter how fine lines appear in an image. Undertone analysis depends on white balance, which depends on the lighting signal extracted first. Inference order and intermediate-feature sharing matter enormously. That insight led to version three, the basis of our production pipeline today.
Dataset construction mattered most
Architecture improvements took us only so far. The bigger gain came from rebuilding the training dataset with photos from actual users across broader conditions, paired with ground-truth skin readings validated by a licensed esthetician in our advisory network.
The work was slow. Good labels require human review of every photo, and trained reviewers agree less often than expected on fine-grained skin attributes. One reviewer may call skin "mild surface dehydration"; another may call it "early barrier compromise." We spent several weeks calibrating the labeling rubric before trusting it enough to run at scale.
The resulting dataset was smaller than those behind most general computer vision benchmarks, but far more representative of our input distribution: everyday consumer photos, diverse skin tones, variable lighting, and the realistic skin conditions we need to detect. At our scale, matching training and deployment distributions matters more than dataset size.
Handling low confidence and graceful degradation
We are glad we built explicit confidence outputs into the pipeline early instead of treating every prediction as equally valid. If image quality is too low for a reliable reading, the pipeline says so. If a particular attribute cannot be assessed from the photo, it returns uncertain rather than forcing a low-confidence result into a hard category.
This matters to recommendations. A result based on six confident, accurate skin-attribute readings behaves differently from one based on six uncertain readings. Matching logic can give confident signals more weight and use broader product families when the reading is incomplete. Users see a recommendation noting which profile details were unclear, plus guidance for taking a better photo if they want a more detailed result.
The pipeline does not produce clinical-grade skin assessments, nor is it designed to. Its target is a set of signals sufficient to narrow a product recommendation meaningfully, not to diagnose skin conditions. A useful recommendation input and a medical measurement are different things, and we try to state clearly which one we provide.
Where it still needs work
The current pipeline performs worst with non-neutral lighting, such as strong warm side light or heavy shadow over one facial zone, and with significantly off-axis photos. Both alter the features used to read surface texture and assess oiliness. We have a partial off-axis mitigation, but it adds latency, and at our current scale we have not decided that the tradeoff is worthwhile.
We are also improving sensitivity-signal extraction. Sensitivity is a skin state, not a fixed attribute, and it correlates with other signals in ways the current architecture captures imperfectly. Early-access feedback shows that recommendations have been least accurate for users with sensitized skin, indicating where the next meaningful accuracy investment belongs.
The current pipeline is useful, but it is not the one we will have in two years. Building it showed us how poorly many computer-vision assumptions generalize to a narrow, consumer-facing domain with uncontrolled inputs. Each failed assumption gave us useful information.