you do not hack files, you hack internet networks. once you are broken through the network you have complete access to every device on it, from there you use exploits in their operating systems to seize control and bypass local permissions. if you are after a specific file and its encrypted no amount of hacking will stop it from being encrypted. you will need to pull the file to your system to begin decryption https://www.khanacademy.org/computing/computer-science/cryptography

a lot depends on how the organization has organized things. A smart security admin will isolate certain systems so no one could have unilateral control from one compromised node I do not really understand public and private keys myself so i will let others explain. https://en.wikipedia.org/wiki/Public-key_cryptography however: Access is different from being able to read the content. you want in to the security vpn you must find a way for it to access it. Brute force, spoofing, exploits ect. the best way is for it to think you are a legitimate user. once you are on there you can intercept the traffic and then you would need to decrypt it. how you do that... if you can get a hold of the keys involved, great simple easy, if not your may have to brute force through an rng, dictionary attacks... crypto is not my specialty :P

the vpn is not encrypted. its content is what is encrypted. and sometimes it won't be encrypted. easy example, Http vs https, anything communicating over http is going to be uncrypted and transmitted in the clear. https interactions are encrypted. however both can be intercepted in route. as for game mechanics, hacking is best handled as high level abstract rather than the minutia. step 1 gain access, step 2 intercept data or find data, step 3 pull out, step 4 find a why to decrypt the data. If you want to use hardware on a network you have gained access to you just use it provided it recognises you have the proper credentials. if not then you need to find a why to byp[ass or override its security. but once you gain access to the device there is no need for crypto

Regarding VPNs and networks... Encrypting your entire network is a tough choice to make. It limits performance, it's a pain to maintain and liable to communications failures, it requires key distribution, and a few smart techies to do everything. What's the gain? If you're on a small ship, probably nothing. The wide gap of space counts as an air gap (or I guess, a vacuum gap :P) The difficulty of getting ON the ship is enough of a defense. For a giant habitat, you'll want to encrypt the portions of your network that require it (probably using VPN) but not the other bits. So the first point is, just because it's secure doesn't mean it's encrypted. Encryption is fantastic when done properly. Doing properly is difficult. There's a lot of encryption techniques with known vulnerabilities, and tools available to exploit them (reducing the time to decryption to days, hours, or seconds, depending on the encryption method and vulnerability). Is this VPN configured properly? If the VPN is connected properly, then it's going to be very, very hard to attack it directly. As Leet pointed out, the easiest point of attack is the end device. If it's not a hardened device (or a security conscious person), normal attacks should suffice. If the VPN is configured properly and the device is configured properly, break one of them. When the tools don't work, there's always an alternate method of giving people access so they can work. The alternate method is frequently (but not always) less secure than the normal method. All of these are unavailable? Attack the user. Really, the user should be the first place you attack. Most famous hacker in the world, Kevin Mitnick, hacked by calling people on the phone and pretending to be a tech. You can do that too. This also does assume you want realism. If you or your GM isn't up to it, there's no shame in just throwing some dice and going with that.