One forum (on old holdens) mentions problems with a "minimum pressure valve" in the master cylinder causing problems with the VH40 booster. Could it be there is something like this in the 404 master cylinder?
AFAIK the disk brake 404 master doesn't have a residual pressure valve.
If the braking system has been working properly at an earlier time then I'd think you are looking at something which has failed due to wear or age.
The master and/or hoses fit that hunch. Check if booster has fluid in it too.