Why some DVLA digital services don't work at night

There might be legal / compliance reasons. It can be incredibly difficult to replace a validated system that is known (or already accepted even if it's technically incorrect) to implement lawmaker dictated behavior.

Otherwise, I think a new approach might be to ignore the specifics of the old system, implement a new system, and a separate translation layer that can run on an export of the old system (or the old system brought back online, but read only after the overnight maintenance) and completely cut over during an otherwise holiday weekend.

That's just data. "Legacy data" was used here to suggest a legacy database/storage system. The reality is that the situation is not due to an insurmountable technical problem but due to a combination of lack of funds / prioritization / motivation / knowledge.

Basically this. The people running the show would desperately like to make it simpler, but ultimately it’s left overly complicated due to priorities from past leadership well above our paygrade.

The right solution is always to just rip off the bandaid and do it again by hand in a new language or platform, and to eliminate useless complexity while doing so. Unfortunately no leader would ever do this because the Board and/or Shareholders would crucify them for not outsourcing it to McKinsey first and using the fancy-pants automation tool their report recommended.

Good for you, it means either your system was sufficiently simple to be fully implemented in 3 months from scratch including all business rules, or you build a new system which left out some amount of rules from the original system without this posing a problem. I don't know much about how central banks work so that might be possible. But not all systems present those characteristics.

For some reason those comments always seem to imply that every business doesn't have these problems too.

Every business has these problems. In most cases, the ones who don't change get swept away. The places that do not change are usually ones that can't go out of business. But every place has systems like this, you have to rebuild them, it isn't fun but there is no choice.

A tiny system like the DVLA is complex, hilarious (this is the same place that has had to reduce service provision because some staff just stopped turning up for weeks after Covid, public-sector productivity in the UK is at the same level as 1997, to just get to the same level as the private sector...which isn't growing productivity very fast...you would need to fire ~2m workers, the total workforce in the UK is 30m btw).

> Think bigger.

One of the easier parts of this involves addressing, which in the UK is notoriously easy, reliable, and easy to process - especially the best-in-class Ordnance Survey stuff like AddressBase Premium, right?

A quick trawl of Github will shed some light on it - especially how much of a pain it is to get ABP into a usable state - and this is data that's core and integral to the service, the "are you a real user, a typo, a fraudster, a data supply issue, or getting things wrong in good or bad faith?" kind of business logic.

And it's doubly hard, because the government requires people to update their license when they change address - which often enough involves a new-build property, where the address (let alone UPRN - sometimes even the USRN!) is completely new to you.

Thinking bigger: imagine sitting at your desk during the first couple of weeks on the job, database validation checks running merrily in the background while you're staring at a screen. There's a mild frown forming on your face. You'd been scrolling over a list of rejected records in front of you, largely looking good - _how did they miss THAT fraud _ you'd briefly chuckled to yourself - but _this_ one...

It's a valid business entity, trading from the valid address, and you've hand-checked both _and_ got a junior who lives nearby to send you a photo of it, and, well, the wit running the business has decided to trade under the name _FUCKOFFEE_, and... that's... just going to have to be someone else's problem, you shrug.

(to be clear: the hard part of the DVLA project is _not_ implementing the coding, database, and systems design work)

Yeah, I always raise an eyebrow at attitudes like that too.

I've also reimplemented or gradually replaced several out-of-date systems. Albeit on a smaller scale.

In my experience, when you start picking the programs apart you find 90% of the code is redundant or boilerplate. Much of it isn't even called from anywhere, abandoned code, and can be deleted en masse. A lot of programmers don't clean code up "just in case" and then no-one else deletes it.

They can also often be vastly simplified because programmers back then didn't have the patterns and knowledge to write consisely.

I often find myself simplifying the original code first, which gets rid of 50% of it. Then I can see what the code actually does and rewrite it which gets rid of the other 40%.

On the other hand, many programmers don't have the patience, stubbornness or skill to do this kind of work.

And the ability to get through the major panic you have when you're half way through and wondering if you were mad to even start.

> Rebuilding a legacy system doesn't require you to support every single edge case that the older system did.

Depends on context.

This isn't some social media fun site where you can live with some rough edges; in this context "edge case" may be someone with an health condition who is still entitled to a drivers license; or it could be someone who normally could get one but due to a health condition really shouldn't be allowed one!

This generally isn’t true in the case of government systems. For the most part they are performing tasks that are required by law, and it is not acceptable to stop some of them, even temporarily. If you’re lucky you can run the old and new systems side-by-side while the 100% feature migration occurs, but that isn’t always feasible.

The point is until you start ripping the application apart you have no idea what the internal logic looks like.

When you look you can find terrors that will haunt you in the night where some ancient limit, say number of columns in a database end up holding multiple structures that are getting if/then'd later in the application.

I would completely and totally believe there is code just like that.

Ya I don't think I'd let you in two miles of a system like this.

Replacing legacy stuff always expands in scope far beyond the initial changes.

When you have to come back and add wait() entries in your new program because it spits data back faster than the old program ever could which then causes peripheral devices/drivers to crash which then pulls a dev and testers off something else important for days figuring out what kind of fresh hell is occurring is just status quo for ancient systems.

> The DVLA dataset and the computations that are run on it can be studied and replicated in 3 months by a competent team. From there it can be improved.

Such an HN comment. Made me lol. Think funnier!

it's a gov agency, they don't quite pay enough for a motivated competent team....

This is what we call cherry-picking stats. You pick a single stat but ignore everything else. Your comment seems to imply that the UK government has been lying about austerity for 10 years. While I don't trust the tory party and think they're corrupt with their deals-for-buddies approach, I don't think they outright lie for 10 years.

The policing budget is so bare-bones that the police have literally admitted they will not attend all 999 calls. To make that clear, they have admitted if you call them in an emergency they may not show up. NHS waiting times are sky-high. The number of NHS hospitals and beds are rock bottom. We can dig and dig into various public sectors and see them being terrible because of austerity. Which Labour is kinda being forced to continue due to the effects of covid on the finincals of the UK (and Brexit)

Public sector employment is at a high because we had to hire thousands of staff due to Brexit… probably the most stupid productivity destroying own goal a country has ever committed (so far)

Are you suggesting that a process once done using pen and paper can't possibly be complicated?

I have no insight into the DVLA, but the idea that no paper process could ever be complicated is really funny. The UK enjoyed/loathed centuries of bureaucracy before computers were invented. At one point getting a divorce required an Act of Parliament specifically naming the unhappy couple! Being restricted to pen and paper hardly inhibited the human ability to create complex systems.

> The system may or may not be complex but the data is has to store and transform is not. Because it handles drivers licenses. A function that has been done on pen and paper and filing cabinets.

It handles more than just driving licenses... The DVLA do more than just driving licenses.

> Since you imply you know more about UK budgets than I do - how much is the DVLA budgeted for IT operations like this and how much more would you give them to expect this problem solved?

It's not budgeted anything for this as far as I know. I believe it's handled by Government Digital Services which handles lots of the digital services for various departments. The budget for all of GDS is about 90 million most of which isn't for .gov.uk. A rewrite of that size I would expect to cost about 50-60 million in total but take several years.

Central government departments have been cut too… the staff needed for the Brexit disaster disguise this

Usually, but it depends on the state. Remember, America isn’t a country, it’s 50 countries in a trenchcoat.

It’s often a mishmash of services too. I was told in-person at the DMV that I couldn’t renew my registration since I’m not the registered owner of my car. So I just went to a DMV kiosk at the local grocery store and did it there without a hassle.

Yes, with a very long list of exceptions which means that many people end up needing to go in person for common situations.

The UK has a similar situation for hospitals. Your driving licence system sounds like a great embarrassment though.

Yeah, as I say. But the other pressures are often temporary and political.

This particular thing might be difficult to change because it's 50 year old COBOL or whatever, but my point was more that I've encountered pushes from architects to "eliminate batches" and it makes no sense. It just means that now I have to re-batch things in my code. The correct way to think about it is that you want smaller, more frequent batches.

Do they really need to do work on all records every night? Probably not. Most people aren't changing their license or vehicle info most days. So the problem is that somewhere they're (conceptually) doing a table scan instead of using an index. That might still be hard to fix, but at least identify the correct problem. Otherwise as you say moving to different tech won't fix it.

Or maybe it is running on faster hardware, but the UK budget office decided not to pay IBM's fees required to make use of the extra speed, so it has been "throttled" to run at the same speed that it ran on the old hardware.

Mainframes have absolutely gotten faster. They're basically small supercomputers.

You're mistaken about this. IBM's z-series had 5GHz CPUs well over a decade ago and they haven't gotten any slower.

It must be. Maintaining the original hardware would be more expensive that upgrading to compatible but faster systems.

There's a very strong network effect, and most stocks are only listed on a single stock exchange, so in most contexts the competition is very minimal.

Bergen county NJ has blue laws that make it so non-grocery stores must be closed on Sunday’s. Maybe there’s some value in structuring a time where everybody is off?

Just like at work the only time I really get off is when all of my customers are off. It’s nice when the industry sorta shuts off for a week or so around christmas

Something like that might plausibly be correct, though you've exaggerated it to a level where it's clearly false.

If we steelman it to its most defensible essence, I think what you're saying is that the cost of the human effort needed to provide these higher uptimes exceeds the consumer benefit (the value of being able to buy a camera on Saturday), say. You could imagine, for example, that each incremental improvement in uptime wins over a proportion of the customer base providing a value that vastly exceeds its cost — but only until your competitors improve their own offering to match, so all the surplus from all this uptime improvement ultimately goes to the consumers, not the producers.

There are two related holes in this idea.

The first is that producing consumer surplus is what the economy is for, in a moral sense. The reason producing goods and services is a good thing to do is so that someone will benefit from using them! So if all the effort that sysadmins make goes into making services better for users, that's a good thing, not a bad thing.

The second is that nothing is stopping a new entrant from offering a new, low-cost service that isn't as reliable. If the cost of providing all that extra reliability (bundled into the incumbents' pricing scheme) is higher than the actual benefit to users, the users will switch to the lower-cost, less-reliable service. This has happened many times, in fact: less-reliable minicomputers stole business from mainframes, less-reliable VoIP stole business from ATM and SONET and SDH, all kinds of less-reliable plastic goods have stolen business from all-metal versions, and now solar panels are stealing business from coal power plants even though solar panel "uptime" is like 30%.

So the particular market dynamics we're talking about actually sensitively optimize the amount of effort given to uptime to the economic optimum. There do exist lots of market failures, but the particular dynamic we're discussing is the opposite extreme from something like a dollar auction.

Who is trying to achieve zero downtime? Facebook has degraded service regularly it's just close enough to 99.9 that nobody cares.

If loading my messages times out I just move onto something else and go back a few minutes later.

Surely they have metrics measuring that and don't think it's worth the engineering effort to improve it.