Indeed Engineering Blog

Use a normal approximation instead?

Miller (2024) recommends analytical confidence intervals using a normal approximation: “we regard bootstrapping as unnecessary unless a complicated sampling scheme or estimator is being used”. We agree with this quote, to some extent. We think both solutions are useful and recommend both, but if a single solution is desired, across diverse estimation tasks, we strongly recommend the bootstrap.

How much does this help?

Offline model quality On an offline apply-prediction benchmark, we compared a strong baseline model with traditional features against the same model augmented with UBM embeddings. Adding more behavior streams improved ROC AUC consistently in the multi-sequence version: Apply sequence only: roughly +3.0% relative ROC AUC. Apply + click sequences: roughly +3.6%. Apply + click + impression sequences: roughly +4.9%.

Why such a sudden change?

The trite answer to the rapid advancement is simply that the models got better, cheaper to train and run (per parameter or token), and more available. That is definitely a factor, but, I believe that more critical advances included: Context Size: The usable context size grew to the point that agents started to accomplish significant tasks with minimal oversight.

What does this mean for the profession of Software Engineering?

I do not think anyone understands the full ramifications of this change. It is too new and too fast. I use the metaphor of going to sleep one night as a blacksmith knowing only hammers and bellows, and waking up the next morning employed at a modern metal shop with hydraulic presses, CNC machines, laser cutters, advanced welding equipment, and even additive manufacturing.

What does this mean for software engineers?

In my experience, this shift is already well underway at many companies.