Monday, October 2, 2023

The Heartbreak of Software

It has become painfully clear in recent decades that, more and more, the major stumbling block in weapon acquisition programs is software, rather than hardware. 
 
The F-35, for example, has failed on the software side of things rather than the hardware.  The all-encompassing, mission planning, parts inventory controlling, maintenance regulating, aircraft health assessing ALIS software has been a colossal, and to date, unrecoverable, failure.  Further, the F-35 was planned to be fully combat capable only with the advent of the Block 4 software upgrade which is now hopelessly behind schedule and has had many of its planned features deleted or deferred to some undetermined future time.
 
The F-35 program, now 4 years into its Block 4 modernization efforts, continues to experience cost increases and schedule expansion. Costs continued to rise during 2021 due to crucial hardware development and testing upgrades, among other things. In 2021, the program office added 3 years to its Block 4 schedule and now expects to extend Block 4 development and delivery into fiscal year 2029 …[2]
As of 2021, the program office now plans to complete Block 4 capability deliveries 3 years later than the original schedule due to software quality issues, funding challenges, and the addition of new capabilities, among others.[2]

If/when it ever happens, Block 4 will introduce over 75 major upgrades into the fighter.
 
So, the Block 4 upgrades enabled by TR-3 are quite imposing, increasing the range and diversity of weapons that can be carried, plus the sensitivity of sensors used to detect, track and engage targets. Most of the new weapons, 17 in all, are “kinetic” weapons such as missiles, but they also include non-kinetic systems that use clever software and waveforms to jam or confuse enemy warfighting systems.
 
Block 4 also enhances networking capability with other tactical systems to enable what the military calls integrated, long-range “kill webs.”[1]

However, before these changes can be implemented, the fighter’s core processor and memory unit must be improved. This is accomplished via an upgrade called Technology Refresh 3, or TR-3. The fighter’s previous computing system, TR-2, is not adequate to support the capability upgrades included in Block 4.
 
TR-3 is being installed in all new production aircraft including the Lot 15 aircraft being delivered today, and will be retrofitted onto fighters already in the fleet back to Lot 10.[1]

This would seem to imply that all aircraft delivered prior to Lot 10 will become ‘orphans’ and relegated to non-combat duties.  Thus, we see that software issues, not hardware, are going to relegate hundreds of aircraft to non-combat status.
 
Software issues not only impose schedule or ‘orphan’ penalties but actual cost penalties, as well.
 
A recent report by the Government Accountability Office noted that the cost of Block 4 upgrades to the F-35 fleet had risen to an estimated $15 billion across over a dozen years.[1]

That’s staggering and that’s just a minor software upgrade cost;  just one block of a planned four.
 
 
Discussion
 
Much of the software delays and problems associated with weapon system acquisitions are self-inflicted in the sense that we create needlessly complex requirements.  For example, the entire faddish trend of wanting to be able to control any weapon from any platform is idiotic in the extreme.  In what real world scenario would this capability be required?  The answer is … none.  This is just the pursuit of technology for its own sake.  If there was no penalty for this it would be fine;  useless but harmless.  However, the reality is that every line of code that is added to a program takes time, costs money, increases the likelihood of unintentional interferences with other bits of code, and requires time to test and debug.  When we’re already struggling – and failing – to produce useable combat aircraft in less than two decades, needlessly complex software is the last thing we should be pursuing.

 
 
 
_______________________________

[1]Forbes website, “Inside Block 4—The Mostly Secret Plan For Making The F-35 Fighter Even More Lethal”, Loren Thompson, 14-Nov-2022,
https://www.forbes.com/sites/lorenthompson/2022/11/14/inside-block-4-the-mostly-secret-plan-for-making-the-f-35-fighter-even-more-lethal/?sh=5172e58f4423
 
[2]Government Accountability Office, “F-35 Joint Strike Fighter Cost Growth and Schedule Delays
Continue”, Apr 2022, GAO-22-105128

23 comments:

  1. Please, bozos running these programs haven't built anything physical in their lives and you expect them to be disciplined and understand software development? You rightly complained about bending metal before the design was complete. Navy SW development doesn't even try to have a design donebefore coding. SW Eng love to say I'll tell you the design after I code it. For weapons and safety critical SW to be developed this way is just negligent. But every one thinks the internet, google, and microsoft examples are the way to go - every read an EULA? Do you want to get on-baord a weapon system that ohas one of those to sign? The Boeing 777 was developed using Ada because they understood that liability, reliability and safety was critical to the sucess of the product. Remember the Mars probe that impacted? That was due to using C with an implied cast that incorrectly converted units. DoD doesn't even require the use of automated code review tools to catch the simplest of programming errors. I program and YES I make errors no matter how careful I am, but I realistically test and try to code defensively. Something SW "Engineers" are not taught at all. If the Navy can't build an oil cooler (DDG-1000), a reduction gear (LCS-1), has to add bouency tanks (LCS-1), can't stablize the 57 mm gun (LCS-2), develop and test a new catapault and recovery system BEFORE installation, how do you expect these political careerists to understand 1s and 0s?

    ReplyDelete
    Replies
    1. All of the aviation world uses the RTCA DO-178 for software development and DO-254 for complex electronics or firmware.

      Both are good, but nothing is better then the people implementing it, hence the B737-MAX f*** up.

      /W

      Delete
  2. I'm a programmer too (but on much more mundane stuff than weapons), and building off John Gault's comment, I can't really say if today's software is needlessly complex, but I can say for sure that no software system will work if you don't follow sound design principles just like no hardware system will work.

    Really, there's no difference at all between hardware and software. The Navy asks for a feature that they don't, themselves, fully understand, and write a blank check to companies that promise to build it and figure out the details as you go.

    One place where software IS different is that it's not understood that the users of a software may need as much training and understanding of how to use it as the users of any piece of hardware.

    You don't put a pilot in an F-35 without huge amounts of training. You don't even give let a guy fire a pistol until he can take it apart and put it back together and fully understands all the parts.

    It's pretty common, though, to give untrained personnel all sorts of software that they only have the vaguest training for.

    ReplyDelete
    Replies
    1. There are two aspects to using software.

      One is the user interface. This seems rather obvious and yet the Navy consistently does not train people to understand the interface. For example, the destroyer collisions involved a significant degree of user interface confusion.

      A second is understanding what the software is doing 'under the hood'. For example, what is the radar discrimination software filtering out and why? The USS Chancellorsville drone strike was an example of the failure to understand what the radar software was actually doing. This understanding is what allows the user to get maximum benefit out of the software. The Navy fails badly (totally) at this.

      Delete
  3. The operating costs for the US big tech companies like Google, Apple, etc. are similar to DoD's entire budget. And the product functions from things like email, search, or phone operating systems aren't all that complex compared to an F-35 or a destroyer. People horribly underestimate how expensive traditional software is to build and maintain. The phone in your pocket takes more resources to maintain than all the Navy fleets put together.

    To think we will get that kind of seamless functionality in complex aircraft with a few hundred copies or in bespoke ships is insane. It absolutely needs to be limited to critical functions like engine timing controls or fire control. Most things should be analog.

    The caveat is that better machine learning is reducing the cost of writing software. But it is nowhere near ready for military applications. The first place we should see it is in cheap end of scout drones (because you wouldn't risk a million dollars in sensors to the algorithm only).

    ReplyDelete
    Replies
    1. The operating costs of Google and Apple are so high because their software has to be compatible with everything out there and their customers expect a lot of drivers and features. The DOD’s Software is more akin to an app, which changes existing software to operate a few known systems, so the DOD has a much simpler design that can be coded by much smaller (think cheaper) teams. Navy needs better project managers who understand how software apps work.

      Delete
    2. "Navy needs better project managers who understand how software apps work."

      You may have a misunderstanding about military software. As a general statement, the Navy does not write software - the manufacturers do.

      There are multiple 'levels' of software:

      1. Equipment software for things such as radar or missiles or electronic warfare sensors. The manufacturer writes all the software for those and the Navy is hands off. In fact, one of the problems the Navy faces is an inability to access source code and make changes. They have to go back to the manufacturer.

      2. Systems software such as Cooperative Engagement/NIFC-C or ship combat (fire control) systems software. This is programmed by the major defense industry players.

      3. Command and Control software. This is contracted out to the big boys such as Raytheon.

      Delete
  4. This also reinforces that an increasing focus on unmanned systems is a recipe for disaster. We can't even get the software to what we want with humans filling in the blanks. We are much farther away from the software doing all of it on its own.

    ReplyDelete
  5. Several things that hinder US Navy SW Development right from the start are the lack of a CONOPS, Lack of user generated USE CASES, and lastly no Computer Resources Life Cycle Management Plan (CRLCMP). The CONOPS and CRLCMP are old proven documents. Use Cases are relatively new and the most powerful SW Design tool that no one uses. Agile is trying to incorporate use cases as User Stories. But the US Navy continues to ignore even these as they clamor for new develoipment "methodologies" that promise everything and delivered tomorrow. On a recent large program award, the contractor delivered the wrong version of the required SW Development Plan and therefore it didn't even address how Agile development would be done even though it was touted as the method to be used. What SW "eng'" or "manager" evaluated that proposal submission. Start bad and you get it delivered even worse .

    ReplyDelete
    Replies
    1. I also work in commercial software development.
      Modern software development methodologies consist in develop first the functionalities that give more value and once the work at specs keep developing the next. And over all DON'T IMPLEMENT UNNECESSARY FUNCTIONALITIES (aka "gold plating").

      If only half of the "gold plating" proclivities of the US Navy you detail in this blog is true F35 software development must be the nightmare of any software manager.

      JM

      Delete
    2. Good point, Gov Weenies usually want everything in the SW even if it doesn't help accomplish the mission. All to let everyone feel like they made a contribution or get something out of it. To me that is why user generated use cases are the perfect check on that. The user can say no I told you what I want it to do. Then it takes a PM with some spine to hold the line. The only exception for overruling the user is legal requirements that they are not aware of.

      Delete
  6. Interestingly is that Israel demanded and got the right to modify the computer code and replace electronics is Israeli made stuff. The F-35 seems to be working for them.

    Oh and the 'Lightning 2' isn't what pilots call the plane its 'Panther'. Just like the F-16 is Viper, F-4 & F-18 are Rhino, and the A-10 is, of course, the Hog.

    ReplyDelete
    Replies
    1. I prefer "Battle Penguin" for the F-35.

      Delete
  7. Why China's J-20 doesn't have same problem? It is not software but people who write software are incompetent.

    ReplyDelete
    Replies
    1. I assume it does have problems. China just doesn’t report it. Do you have inside information showing that the aircraft has been tested and found to be free of problems? If so, please share the results with us! If not, you’re engaging in unfounded speculation.

      Delete
  8. "the entire faddish trend of wanting to be able to control any weapon from any platform"
    Not just any platform, but from a platform that is not carrying it. That is the whole "distribute" nonsense. The idea that one ship (the one carrying the Admiral I assume) can fire the missiles off a drone or escort ship.
    Just like the love affair with drones, I believe this has less to do with better tech than in the intense desire to micromanage by the top brass and their civilian counterparts.
    Mid rank officers are already terrified enough about daring to disagree with their superiors and actually say "yea, we can't do that because my crew is exhausted, my ship is rusting, and the system is buggy."
    Now they can bypass all that. They can ignore ship's captains entirely. The orders in combat will be "you sail here and I will push the buttons."
    Drones? Drones don't get to testify in a court martial that "I warned my superiors that the fire control system wasn't working but was told to sign off anyway." or "We have never actually fired the weapon system at a moving target".

    ReplyDelete
  9. Being a Software Engineer, I am able to related to this issue
    as famously quote "One of the biggest traps for smart engineers is optimizing something that shouldn't exist."
    I read somewhere that F-35 in its inception was suppose to be a Light Bomber/Heavy Recon Aircraft
    however, as time goes on, various new roles were assigned to it.

    #ComNavOps, do you know why US Pacific Forces operates F-35B ?
    operating F-35B in Europe against Russia is OK, but operating it from islands spread across vast Pacific Ocean doesn't make sense
    Especially when F-35B has almost half the payload for almost half the range when compared to F-35C

    ReplyDelete
    Replies
    1. "do you know why US Pacific Forces operates F-35B"

      I'm not sure exactly what you're referring to.

      LHx ships can only operate the B model. Carriers only operate the C.

      The Marines fantasy of small hidden bases precludes the C model due to the need for large, long runways.

      We lack bases to operate the A or C models in the Pacific.

      As I said, I'm not sure what you're really asking. Try again?

      Delete
  10. https://www.msn.com/en-us/news/world/fifty-five-chinese-sailors-are-feared-dead-after-their-nuclear-submarine-got-caught-in-a-trap-intended-to-ensnare-british-sub-surface-vessels-in-the-yellow-sea/ar-AA1hDPQU?ocid=hpmsn&cvid=efe72085e5d644b9a515813696f37b8f&ei=8

    Obviously China isn't saying anything and full denial, feel sorry if true for the sub crew, sucks to die like that.

    Note was the sub got snared in some sort of anti underwater drone trap....never heard of that before. Heard about nets around ships, nets or chains to deter entrance to ports but this sounds different.

    ReplyDelete
  11. I wonder if there's a sort of Dutch Disease going on with software. That's where one field (like oil) becomes so profitable it stops everything else in the economy from developing.

    For the US, we are blessed with a world-leading software industry. A talented programmer can make big money at all sorts of tech companies here. But that also means the military has to pay big money to compete. And they... don't. So it's really hard for them to recruit good programmers.

    ReplyDelete
    Replies
    1. "it's really hard for them to recruit good programmers."

      You may have a misunderstanding about programming. As a general statement, the military doesn't program. They don't write code. The manufacturer (of, say, a radar) writes the code as part of the product. If changes are needed, the military issues a purchase/change order to the manufacturer to make the code modifications.

      Delete
  12. "Lockheed CEO Pitches Pentagon on Subscription Software"

    Now to fly the F-35 you have to have valid credit card entered
    before takeoff ?
    Story is at USNI, LM must read CNO, decided to prank him.

    ReplyDelete
    Replies
    1. Do you have any subscription service that doesn't constantly raise prices, year after year, once they've hooked you? I don't.

      If Lockheed's CEO is pushing this we can be 100% sure of one thing: it's good for Lockheed.

      Delete

Comments will be moderated for posts older than 7 days in order to reduce spam.