Thursday, March 21, 2019

Stadia is Not "THE" Future of Gaming

Thoughts on Google Stadia From a Game Loving Cloud Compute Guy

As I slogged through the marketing grandstanding of Google’s Stadia GDC live stream, I grumbled and cursed many a times while making my tender lamb stew. I love gaming and I love cloud computing, so why should Stadia- an “innovative” new service capable of running and streaming GPU intensive games from Google’s cloud servers directly onto any device of my choice cause me such consternation? We have grand declarations periodically that “X is the future of gaming” with X being VR, mobile, on-line multiplayer, and most recently and pitifully ray tracing. No one took Jensen Huang too seriously and ray tracing is definitely not the holy grail of gaming. VR is still struggling and single player most definitely wasn’t killed off by on-line shooters.

If there is one thing that the brief and glorious history of video gaming taught us is that gaming is endlessly diverging not converging. Stadia has to compete ferociously for it’s life to coexist in a rich heterogeneous ecosystem that includes consoles, PCs, modding, on-line shooters, all those grindy MMOs, grindy mobile games, VR/AR games, other competing streaming services, retro games, emulated games and more. The vast swathes of humankind’s gaming catalog will never be running on Stadia servers. Streaming is not going to drive locally running games to extinction just as chromebook did not kill the PC which instead highlighted limitations of network only applications.

As a VR/RPG/modding gaming enthusiast who has spent significant time wrangling on Google’s Cloud Platform GCP as well as on Amazon’s AWS and lastly on Azure, I thought I should put more meat on my instinctual skepticism about Stadia and pure cloud based streaming models in general. While I agree with the usual howls of network latency killing gameplay from first hand experience, Stadia has many more challenges to overcome in generating enough of worthwhile content given the hurdles of porting. Also against common agreement, I also don’t believe it’s a complete given that Google has the cloud part in the bag as there are challenges to implement cloud gaming efficiently in managing beefy GPU hardware capacity that is less scalable. How can Google make the pricing attractive and profitable for a potentially laggy and subpar gaming experience that isn’t geared towards serious gamers?

When Google staked a very public claim in streaming with the GDC announcement, I’m sure Microsoft/Sony/Nvidia/Valve people must have had a gut check- irritation and fear in different measures but gladdened Google only bloviated with grand statements backed up only with a lackluster demo. The Stadia demo of Assassin’s Creed soaring above the landscape was cleverly picked which hides any lagginess instead of a first person shooter demo. Moving the game between devices was old hat for those that have already seen Blade’s Shadow demos a year ago. For those not in the know, Shadow is a cloud streaming service that provides a cloud instance with a dedicated GPU running Microsoft Windows- a glorified remote desktop of sorts that you can move around in different devices. I’m sure all the Shadow engineers blanched the most at the web giant’s power move. I have to confess my initial hate on streaming stems from my negative experience last year with Shadow-mainly due to my subpar laggy internet service.

Let’s say for the sake of argument that latency is not an issue. Any gaming platform’s strength is in massive adoption. What will make the masses with fast internet and ready cash flock to Stadia? The driving forces for adoption have historically been gaming content/experience and cost and I don’t see it working out in Google’s favor in either front.

Discomforting Loss of Gamer Control in a Streaming Model

Even before talking about content or cost, streaming represents a disturbing trade of convenience over control. Many gamers on YouTube and reddit loathe and fear streaming as a way corporations grab control. Google aims to control how you play and when you play and serious gamers are right to be weary of the dark sides to this trade. This streaming service may appeal to uninvested casual gamers who want a way to pass the time, but for serious gamers that tend to spend hundreds of hours in single player games, giving up such control is terrifying as the game is more than a game but a player’s private world especially if it’s been modded.

“Netflix of Games” is a readily consumable analogy but trivializing the gap between streaming static content and a video game while disrespecting the dynamic interactive complexity of gaming as a fundamentally different medium. Phil Harrison, head of Stadia, touts the film/music analogy as a way to wave off those weary of giving up ownership but note the biggest blaring difference- the significant majority of films/music have been fully digitized and are largely available in subscription services or for an extra fee. Most games that require hefty GPU horsepower are too complex to automatically convert and 99.99% of the existing work will certainly NOT be converted to this Google platform. From what little Stadia unveiled, it excludes too much and provides too little currently to become a major force in gaming.

Stadia Pricing Based on Existing Cloud Pricing
Current cloud costs of an instance with hefty GPUs are expensive. Since Google filled out an hour without any mention of the pricing model and have kept coyly mum in interviews, we only have our guesses on how they might monetize the key costs- hardware (CPU/GPU/Disk) vs game title. In looking at existing cloud pricing models, obvious possibilities include metered usage and/or a subscription service with neither precluding the other along with game licensing and micro transaction costs.

Projecting Stadia Pricing Based on Existing Cloud Pricing

Current cloud costs of an instance with hefty GPUs are expensive. Since Google filled out an hour without any mention of the pricing model and have kept coyly mum in interviews, we only have our guesses on how they might monetize the key costs- hardware (CPU/GPU/Disk) vs game title.  In looking at existing cloud pricing models, obvious possibilities include metered usage and/or a subscription service with neither precluding the other along with game licensing and micro transaction costs.

These beefy specs don't come cheap. Also no mention of disk...
Standard cloud services generally provide metered billing(per second) for the underlying compute resources with either additional metered cost for software license such as Microsoft Windows or allow the user to bring their own licenses. Looking at existing GCP cloud pricing for a single GPU, the closest spec to Stadia would be the NVidia Tesla P100 with 16GB HBM currently at $1.46 per hour not counting storage costs. This cost is expensive even by cloud standards since high end GPUs are expensive though Google would have squeezed a much better deal out of AMD providing the GPUs. Even $1 an hour would be much too steep for a casual gamer and underlying costs are probably too expensive to provide a free to play model. Shadow currently has a subscription model for $34.95/month which I think is too much of an outlay for the casual gamer but most likely a profit loss if the gamer plays too much. Metered usage opens more wallets than a monthly subscription. Any giant creating this type of gaming platform has to be ready to lose billions for years on end and you have to wonder how long Google is willing to persevere with full commitment given their history of starving projects and unceremoniously ditching failures. Gaming is not a part of the Google DNA. If I had to bet, I would bet more on xCloud succeeding than Stadia. You have to give credit to Microsoft's continued perseverance to gaming despite stumbles- the first Xbox was 17 years ago- 2001.

How would Google manage hardware capacity and demand efficiently?  Unlike other general use cloud platform where vendors have more wiggle room in slightly over-provisioning to accommodate more users, Stadia will have hard limits how many users you can fit into a data center since currently a GPU cannot be split across user instances(if I am wrong please correct me).  Stadia's predominant usage is driven by a live human clicking behind an instance which has to be of close geographical proximity to the user to reduce latency at concentrated peak hours.  You can't move users to a different region without risking worsening latency. If launching Stadia is as easy and casual as clicking a button on youtube to play a game, how well can Google predict peak demand and grow the data center.  It takes time to add new machines- in the meanwhile users lose confidence being locked out when paying for a premium service. Google will have to carefully limit users to prevent a stampede, but what will they do with spare capacity if the masses don't come?

What will Google do with the spare capacity in off hours(working/sleeping)? Run their own machine learning algorithms for world domination? By gods, is Stadia one of Google's evil plan to have more machine learning capacity.  They will probably run ML processes that crunch your gaming habits to see what ads to serve you.  Even if Stadia loses money, Google still would have scraped a lot of data they couldn't get any other way. Google could also sell off Stadia's spare capacity for AI processing just as Google's cloud service sells spare capacity with preemptible instances which get used for short-lived batch processing?


Input response time is not the only performance concern.  Startup and load even locally can take 10+ seconds to a minute depending on game content and disk media.  On Google's cloud,  I can vouch for how fast base Linux instances can be spun up (5-10 seconds) but launching game executable from scratch requiring gigabytes to be read may take significantly longer on the cloud than locally depending on implementation. Verily Google could have pre-warmed instances for the most popular games to speed startup but most GPU heavy titles dynamically load gigabytes of texture maps and worlds depending on gamer location. An important takeaway for those that have not used a cloud service, one pays through the nose for faster disk access on the cloud which is always slower than local versions of the same type.

 The bane of my big data cloud existence was expensive limited slow storage even in SSD across all vendors though GCP was better than Azure for persistent disk.  Disk access is capped by throughput and number of IOPS or reads and writes per second based on disk size.  In GCP, you would need 1TB to get 500MB/sec at a hurtful $170/month and with dozens of servers- storage costs were a constant bugbear we had to manage.  Granted for Stadia, Google would segregate stateful game data and avoid long term storage for an instance of the base game to bring down storage costs but  how would they manage disk performance running predominantly IO hungry games.    The reason cloud vendors can get away with shared SSDs is that not every tenant on a server is banging on the disk with equal ferocity.  But modern triple AAA game can easily be 50GB-100GB with gigabytes of texture maps and worlds being loaded dynamically but perhaps Google is banking on the fact disk usage spikes are intermittent.

Compelling Gaming Content and Experience

In the varied ways a games can hook users, the best examples of the genre strive to be genuinely engaging and worthy of your time (Witcher 3, Kingdom Come Delivernace, too many titles to list!!!). But the rise of mobile gaming unleashed a rapacious insidious trend of employing psychological manipulations such as intermittent rewards to bleed tiny drops of money out of you. To the detriment of the gaming genre, it’s easier and more lucrative to create Candy Crush than labor 7 years for Read Dead Redemption II. Mobile games can adopt a free to play model to lure casual players but clearly the Stadia hardware is much too expensive to allow users free entry even if the game license is free. Stadia would have to go down partially the harder route of compelling content. Stadia has started their own studio but can they hit one out of the park? Creating a hit game is not something you can just throw money at. Ask Microsoft. And Amazon.

Only recently I understood the reason 91+ million Sony PS4s blot this earth. I had been a PC gamer most of my life and had no desire to downgrade my 4K GTX 1080ti gaming experience with a measly console. But Red Dead Redemption II only available on console made me finally capitulate and shell out $500 for a PS4 Pro with fake 4k which I grudgingly bought over an XBox One X which had superior hardware. All the console games I wanted to play(God of War, Spiderman, Horizon Zero Dawn, Persona 5) were Sony exclusives- a winning trend Sony is not going to give up. Although the hardware by the sound of the fans struggled mightily on my PS4 Pro, I was utterly shocked at the superior graphics of RDRII pushed out with a mere 4.2 teraflops, only a third of my GTX 1080ti. The breathtaking scenery Rockstar created by black magic exceeded cinema. I don’t think I’ll be consigning this PS4 Pro to the dustbin any time soon as such a rich library still awaits to be played. Existing consoles are such a formidable competitor to streaming, even Sony have to compete with themselves to convert their own existing user base to new services. Stadia would have to provide something better and different.

Double Rainbow of Red Dead

Google only announced Doom Eternal, Tomb Raider, and a handful of other titles as a confirmed launch title besides Assassin’s Creed Odyssey- a roster not strong enough to open the wallets of the masses. Doom historically has had low specs that you can play on an aging potato PC. Since Doom is not an exclusive deal, unless Stadia has an attractive value proposition where it’s significantly cheaper to try than buy, I’m not sure all the Youtube integration and next gen features would be enough. Splitting distribution this way risks cannibalizing publisher profits, it’s a huge gamble on Bethesda’s part that more people that would not have bought a copy will play. I could see that it’s good for the gamer if Stadia becomes an easy/cheap way to try games before you buy. Given the current trend of buggy first releases of AAA titles, publishers will probably try to circumvent any disadvantage. If Fallout 76/Anthem had debuted on Stadia in their wildly buggy states without an upfront license, undoubtedly publishers would have lost revenue.

There is a steep cost to the game developer to bring a game to Stadia who are already too stretched. Beyond the major obvious difference of creating an executable that runs on Linux with Vulkan, running games efficiently on Stadia is more than a matter of running the same game executable you can run locally vs in their data center. Even if Unity and Unreal Engine will “support” Stadia deployment, it’s probably nominal alpha(buggy) support initially but the bulk of GPU heavy games are built on custom engines.

On the positive side, I’m thrilled with Google’s choice of Linux/Vulkan. Even if Stadia becomes one more tombstone in the cluttered Google graveyard, Linux gaming got an unimaginable boost. The current list of Vulkan supported Linux games is … well I rolled through that stunted list multiple times in the last few years without being enticed once. The second positive silver lining- Google spends some of their search engine billions on furthering gaming. Google being exceedingly late to a crowded field is predictably is looking to disrupt the gaming industry with a completely different cloud streaming paradigm integrated with a broadcasting platform. Shouldn’t we welcome innovation and competition which in the end is good for the end gamer? Or will this be wasted divided effort on developers that release on Stadia and other platforms. If Doom sucks, the web trolls will come out and blame time wasted on Stadia.

But a greater harm to gaming would be that if streaming as the predominant model would impact the type of games that get made. Mobile platforms already generate enough insidious free to play microtransaction laden Skinner boxes, but at least their attempts are obvious. If Google and publishers have complete control and knowledge of all your gaming inputs along with all Google has already stored about you, they could with greater subtlety inject their attempts at squeezing cash out of you in and out of game.

Final Thoughts on the Future of Gaming

I am heartily convinced console, PC and many other forms gaming won’t be supplanted by streaming any time soon if ever. Inadequate internet infrastructure and data caps are not the only obvious drag on streaming adoption. Even with 5G and fiber, network latency is a factor that won’t go away for VR where latency=nausea and in competitive games where latency=loser. How can a single new untested platform dominate the future of gaming when the massive existing and growing catalog of fantastic superior games are available elsewhere? Just as evil people routinely predict the demise of single player games, it is a hot steaming pile of cloud conceit that streaming is “THE” future of gaming. History has favored divergence and not convergence and what kind of poor diminished world would we gamers live in if we were reduced to streaming as our play mechanism.

Friday, February 1, 2019

My Adventures in Performance Tuning Fallout 4 VR

TLDR;- FO4VR still takes serious tuning to make it playable even in 2019 even for a decent rig. FO4VR is a CPU intensive game and sadly it was upgrading to an i9-9900K that made the biggest difference. But tweaking and modding can bring FO4VR to an "acceptable" state.  
When I first started playing Fallout 4 VR on my Alienware R4 17 with an external GTX 1080ti,  the stuttering and blurriness was so atrocious, plays would give me severe eyestrain then headaches.  I've invested a significant amount of time trying to bring FO4VR to the level of my heavily modded SkyrimVR visuals but I gave up and tried frantically to stabilize with acceptable graphics.  After I had given up everything(shadows,interior lights,particle effects,grass) to run a decent stable game, I realized I couldn't do any serious settlement building which was a big part of my enjoyment.  So I finally capitulated and built me a new gaming rig.  However I decided to chronicle my performance tuning in detail as I still needed all the fixes even for my beefy rig to load up on 4k textures and to mod further.

The bad performance in this game is a death by a thousand cuts and there is no magic bullet- even an i9-9900K/GTX 1080ti requires compromises and tuning. I still won't run lush tree and grass/ hair physics mods since my reprojection rate can run as high as 30% in my growing settlements. My nightly/weekend tuning exercise fell under 5 categories:
  • Performance Mods- Probably Lights Remover made the most noticeable difference.
  • Fallout 4 VR config/ini - TAA tweaking, lowering shadow quality, particle effect, turning off grass, etc
  • SteamVR Configuration- Supersampling and reprojection options. Before my hardware upgrade, I ended up running "Always On" interleaved projection with  Motion Smoothing to reduce stutters.
  • Existing Hardware Configuration such as overclocking GPU/CPU/DRAM, GPU power options
  • Hardware Upgrades- unfortunately the most expensive yet the most effective
Also the performance problems vary in nature- low fps, stuttering, high load times,  slow and unstable workshop menu. I've put more details below this summary diagram.  Even though my husband thinks this diagram is overkill, I use this diagram to reason more systematically about my tuning. The green to red color scheme indicates performance impact positive to negative:
  • green- improved performance significantly 
  • light green - moderately improved
  • white- setting at neutral (could crank up)
  • yellow - nibbles away at performance
  • orange- degrades performances significantly, but I use the settings to improve immersion
  • red- kills performance and hence avoiding them for the present
  • blue- not sure if it worked as a FPS boost was not seen


Performance Tuning Tools- FPSvr

A good performance tool can more objectively show the impact of your changes and I highly highly recommend FPSvr. Only for 4 bucks you get a GPU/CPU breakdown for each session that can show you where you bottlenecks are. (It's sad that SteamVR fps tool is so unusable.)  You can check  the FPS rates and CPU/GPU usage in-game by tilting up your controller, and at the end of the game session stats are automatically saved to analyze through the included FPSvr Viewer.  Having historical baselines for comparison is huge when you are piling on mods at every play.

Click me to see this fantastic tool for VR session breakdown.

If you click to enlarge, you can see the breakdown of GPU/CPU frame times and usage. I used this to see that the CPU was no longer a bottleneck and the current Intel Turbo Boost (~4.6GHz -~4.9GHz) was more than ample for this game and I didn't need to bother overclocking just yet. In my old rig, couldn't play without overclocking my i7-6820HK.

Yes you can see from the sessions table on the left that I played an unhealthy amount of  FO4VR this week except this wasn't exactly playing- I was performance testing all over the Commonwealth on my new gaming rig obsessively looking at the in-game FPS monitor and checking the newly installed 4K textures.  

In addition, I also use GPU-Z and HWInfo  to monitor usage and temps to see if I can eke out more from the hardware.

Performance Mods

I installed all the standard recommended performance mods but it was only the Lights Remover that I saw a noticeable difference. Admittedly I did not have as much discipline in my early testing without FPSvr and was relying on SteamVR in-game fpsviewer.
    Insignificant Objects Remover, FAR, OVT on their own did not fix my initial stuttering problem. However I leave them.
Mods that I couldn't yet see an improvement:
  • Load Accelerator - I really really wanted this mod to work. I still have slow load times(15-30 seconds) on my beefy rig and m.2 Nvme SSD.  Performance stats show CPU/GPU are just hanging out not working much during this time. Curse you Bethesda- you still found a way to defeat me.  I am giving up on this at the present since I spend the time looking at my FPSvr stats and resting my eyes.  
  • Boston FPS Fix - this curiously increased my FPS in Boston areas and I don't know if I have a conflicting mod that undid this mod's changes or the weather changed.  I need to spend more time retesting.

Increasing Workshop Menu Performance

When you mod in thousands of items to your workshop menu, you are going to start seeing the workshop menu load slower and slower.  There is a fix for pancake involving F4SE which sadly will not work in VR.  Surprisingly turning off the rotation animation in your Fallout4Custom.ini does provide a performance boost and prevented my games from crashing every 10 minutes to about 60. 
[Interface] fWorkshopRotateSpeed=0.00 ;(Default: 1.00) fWorkshopRepeatTime=0.15 ;(Default: 0.25)
I found the solution from this handy reddit thread:
https://www.reddit.com/r/fo4/comments/4panl5/psa_you_can_stop_items_from_rotating_in_the/

Moderating Mod Usage and Avoiding Performance Killers

As soon as I get my build stable, the first thing I do is slap on more mods.  I am greedy when it comes to mods and I know "some" of my performance/stability issues stem from my mod addiction. (The terrible initial performance, blurriness and stuttering was ALL Bethesda since I had it before modding!).  Mods are what I love about Fallout and this habit requires a lot of restraint.

I can not NOT have 4K texture mods so I make a tradeoff and pare everything else down.  I try not use any mods that turn off or mess with precombined meshes no matter how hard I want to get rid of garbage in my settlements. See these explanations from: 
Then there are cases where my own gameplay rather than the mod degrades performance. For instance Sim Settlements is an excellent mod is not a performance killer in itself.  However it does make growing your settlement so easy that I tend to create larger settlements than I normally would if I had to micromanage it all by myself.  Fallout 4 settlement building in general is a performance killer and you can see many youtubers (Oxhorn) talk about how frame rates drop below 30 in their heaviest settlements. That's bearable in the pancake version but totally a no go in VR.

Unfortunately this poorly optimized game may force you to change your gaming behavior. I don't build settlements in a heavy areas like Hangman's Alley or Jamaica Plains and prefer those situated in decent performing areas.

Fallout 4 INI Tweaks, TAA etc.

Tuning TAA-  I tried turning TAA off, using FXAA. FXAA just hurt my eyes and decided I must have TAA. You have to tradeoff between sharpness and sparkling edges vs blurriness- sharper it is more sparkly it gets. Shine like a diamond, shine like a diamond in the sky~~~ My current TAA settings are for 2.0X SS:
fTAAPostSharpen=0.75
fTAASharpen=1.0000
fTAAHighFreq=0.8000
fTAALowFreq=0.5000
fTAAPostOverlay=0.75
I used lower 0.675 for  fTAASharpen/fTAAPostOverlay when I used to run at 1.5X SS.  I adjust shadow settings, ambient occlusion all in-game.

SteamVR Configuration, SS etc.

Managing Reprojection

With my old rig(i7-6820HK/1080ti), pegging at 45 fps with the  "Always On" interleaved projection Application setting was THE only way to reduce stuttering by reducing overall load on the CPU which would run below 80% and have headroom. If I ran without it (with just motion smoothing/ asynchronous projection), as soon as I went into a heavy area like Boston or Sanctuary Hills, CPU would run up to 100% and stuttering would ensue. Yes combat of any kind esp. heavy shootouts suck with interleaved projection but it beats getting a headache. But now even with the i9-9900K, I have motion smoothing on and reprojection rates still range from 5% to 30% depending on how fast I whip around and which area I am in. 

Supersampling 

I manage super sampling strictly from SteamVR Application Menu. I run at SSx200% and I'm surprised how much clearer everything is- esp. when reading text in game.  I know this is the knob I can crank down for more performance if I need to but I am loath to give this up. Supersampling taxes the CPU and GPU both. I could only manage about 1.5x with a i7-6820HK for CPU load but I still used to get the stuttering problem at 1xSS.

Performance Goals and Tradeoffs

In PC gaming, the more you tune and tinker, the less time you leave for gaming. Unfortunately FO4VR is like an old Alfa Romeo.  It is not a game you can simply pay higher components to buy yourself out of performance problems due to the creaking engine and modding.  I still have slow loading times and workshop/settlement problems that are barely utilizing the underlying hardware. Sadly with FO4VR, I find myself  trading off game enjoyment with with fixing performance problems. There's how much time you will spend to get things better and there's also what you have to sacrifice in order to get there.

You might just be trying hard to stabilize your FO4VR and reduce stuttering trying anything possible.  Once you get your system stable and depending on your system's capabilities,  you might have choices on what to cut instead of having to cut as much as possible. In the old rig, I cut out everything- no grass, low shadows, no interior light, no face light, no particle effects, and peg to 45fps which wasn't great during shootouts but this let me start building. But when I heard from some youtubers that they can only play their big settlements  20-30fps, I realized settlement building was a ticking time bomb to unplayability for VR. That's when I bit the bullet to build a new system. 






Wednesday, January 30, 2019

DIY Custom Gaming Rig: Part II Building

The night before, I made myself  a detailed step by step plan on how I would iteratively install and test the components.  I went to bed overexcited and a tad slightly apprehensive about applying the thermal paste.

Once all the components rolled in from Amazon, I cleared off the dining room table for starting the build. Unboxing raw components definitely had a excitement I haven't felt with fully formed consumer electronics like a PS4 pro. When did RAM get so big and beefy? After years of laptops, it's fun to handle a motherboard again. I was impressed with how much was crammed into this MSI z390 motherboard except for the dragon decal is completely unnecessary and gratuitous. The i9-9900K was smaller and thinner than a triscuit.  Hooking up sockets and power cables was intuitive. The most fun part was popping in the DRAM and seating the core i9-9900K in the LGA1151 socket.   

Everything from Amazon had arrived with 1 day shipping except the higher grade Kryonaut Thermal Grease which wasn't due to arrive a day later.  My plan was to use the included Noctua thermal paste, do testing and then repaste with Kryonaut the second day if temps were not satisfactory. 

I had watched many a youtube video with the mesmerizing application of grey goop with my favorite being from Gamers Nexus which reassured me that technique did not matter that much as long as you didn't squirt an obnoxious amount.  I really really wished I had watched JayZTwoCents video on how to build a PC the wrong way before I did this. Even to his surprise, doing all the wrong things like just smearing on gobs of thermal paste with his finger, separating the cooling sink and reattaching actually did not affect the resulting temps that much. I was totally OC (not overclock, obsessive/compulsive) about a great many things that did not matter in the end and was tripped up by things that I didn't expect.

Before I could power on the system, I had to install the CPU fan. The absolute worst(THE WORST) part was installing the Noctua fans which took a ridiculous 5 pasting attempts mainly due to the high standoffs. The whole process of reinstalling involved having to unscrew the mounting frame to unseat the cpu which caused the backplate to fall off, popping off the cpu, cleaning cpu and the heat sink, re installing the backplate which required flipping case on it's side and re mounting frame, seating the cpu, squeezing out some thermal paste, pressing down the heat sink. Also you had to hook and unhook the two fans if you had done a full install. It took half a box of coffee filters with isopropyl alcohol to scrub out a wasted $10 worth of thermal paste.

Day 1: A fully functioning PC with  Noctua NT-H1 Thermal Paste, Horizontal fan placement


Attempt 1: I followed the Noctua instructions to the letter and put a small 3mm pea of the included Noctua NT-H1 thermal paste as instructed. I press down but I have this weird feeling it's too little. I pop it open to see an uneven oval. (It's entirely possible that it would have covered everything once it was screwed down....)

Attempt 2: I clean everything up, follow the steps again with a more erm generous helping of thermal paste. However the second screw will not catch at all to the standoff.  My husband who conveniently came home for lunch exerts his large hands over the mobo. He asks me to cleanup everything and so he can test the fit and debug without the thermal compound. He keeps asking me if I didn't seat the CPU right and so it's sitting higher. I keep insisting it's impossible- this LGA socket is idiot proof.
Cleaned up this cold plate 4 times,
 3 times too many.

    • Attempt 3:  I clean everything up again and show him how to pop in the CPU in the socket. My husband thinks the provided Noctua standoffs are too tall and the screws will not catch even with what he considers reasonable pressure. Since he's a precision machinist by training and he knows a lot about screwing in things, I take his word. We try the other shorter standoffs included for the AMD chip. (According to web board chatter, you have to use an unnatural "sketchy" amount of pressure. It's kind of scary to press down that hard because you don't want to crack the motherboard.  Since I have Spongebob Squarepants hands, I was happy to be able to use this shorter stand.)  My husband squeezes out the very last of the thermal paste but there's really not enough left in the tube but we screw everything in. The Kryonaut is not coming until the next day and I wanted to do benchmarks anyway to see the impact of too little paste.  Thermals are not great but in acceptable  range. Also I also did a horizontal fan placement like a hamburger parallel to the GPU to see if I can work out a different airflow as this fan was so unwieldy, it didn't quite fit blades hitting against the mobo 24 pin power cable.
But by mid-afternoon, I had a running computer that had Fallout 4 VR running on it smoother than I ever had seen it.

Day 2:  Kryonaut Arrives, Vertical fan placement, HDD Install

I guess all those Grizzly Kryonaut commercials on Gamers Nexus and Linus Tech Tips must have worked subconsciously into my brain as I didn't think twice about that.  I won't use liquid metal because it's too nerve wracking and after having cleaned the cpu and cold plate 4 times, I realize the stuff gets everywhere even when you think you are being super careful.
Prettier without them flesh colored fans.

Attempt 4: I open up the heat sink and yesterday's last squeeze was definitely was not enough. I clean it up again and apply the Kryonaut with the included tiny paddle but it's too thick and sticky and clumps up all the wrong places making me worry I'm just whipping in more air pockets in an effort to spread it.  I decide to start over and use the X method instead. 

Attempt 5: I unscrew the mounting bars one last time, unseat the i9 one last time, clean both surfaces not as obsessively as yesterday, reinstall the backplate, and mounting frame one last time, totally forgot I want to use an X, and just plop down a fat dollop of Kryonaut goodness. I just screw everything in, do a benchmark and see pretty good thermals(10+F decrease from yesterday) and call it a day.  I now hate and love this Noctua fan since it's quiet and performs admirably.

Day 3 Fan Readjustment Causes Ugly Cable Syndrome  

I had put the fans to left of each tower as placement to the right would causing the blade to run into the motherboard power cable. The old configuration wasn't optimum having the back case fan be back to back with a Noctua fan. I shifted the fans to the right over the memory slots which forced me to pull out the power cable over the top of the stylish blue cable bar. NZXT has gotten some flack over the cable bar being too high and just 2 centimeters lower would have solved all my problems and now my cabling job looks subpar and prevents me from posting on Instagram.  But the thermals are cooler than ever (idle and under load) and so performance over looks wins out. (Also I was not going to unhooking and rehook these fans yet again- the fins are sharp and fingers were all cut up.)
FINAL fan positioning forcing power cable in front of bar
But now I can finally close up the machine, do a closed case thermal test to compare against the open case test.  The machine can now graduate  from the dining table to it's rolling cart into the living room. 




The RGB configuration is for temps, not for aesthetic purposes although it's a pretty night show completely hiding those flesh colored Noctua fans.

Upgrading BIOS, Resetting CMOS

On the first day when I installed the GPU, I changed the graphics card setting to PEG and hence going BIOS would just end up as a black screen. I could have debugged this on the first day but decided to do all the VR software install since everything else was working. 

Now it was time to start overclocking which meant I had to reset the CMOS which required moving the GPU out of the PCIe slot since a fresh MSI BIOS will not boot with the GTX 1080ti in the slot. 
I also wanted to flash the latest BIOS . Since MSI doesn't ship with a jumper cable, I had to use a flathead screwdriver to short the CMOS which didn't make me feel comfortable but was fine.  

I could service everything vertical on the cart but I didn't feel comfortable handling my heavy 1080ti hanging off the slot(I have the dread GPU sag of less than a cm) and ended up lugging the case back out on the floor.  But now I know intimately the layout of my rig and got familiar with the MSI BIOS. It just needed another graphics setting.


Monday, January 28, 2019

DIY Custom Gaming Rig: Part I Specs

Friends, it was finally time to build my very own! I've always bought laptops making DIY unpractical but now due to my increasing (ahem) compute needs, I was ready to commit to a stationary beast of a PC tower. My Alienware 17 R4 laptop which had served me so well for 2.5 years was now limping along due to an aging CPU i7-6820HK even though it had plenty of GPU power of a GTX 1080ti with an external GPU chassis.

Truth be told, my laptop is perfectly adequate for all titles except for Fallout 4 VR- a game so poorly optimized that it needs a 8 core CPU released years after the original game to run full graphics. Right now I have to peg to 45 fps, reduce AA and supersampling, reduce load distances and shadows, turn off dynamic grass, turn off interior lighting and face lighting, turn off rain and radiation storms and dozens of performance mods to run stable. It hurts my eyeballs when I play and my vision is my one deteriorating resource.  It's not quite potato mode but this was not a game I could currently give friends a VR tour even though it was the only game that contained my very own structural creations. And of course I could not even think of upgrading to a next generation VR headsets without a beefier machine.


When you go all custom, you have all the freedom in the world which translates into making dozens of micro-decisions over hundreds of competing items then cross checking for compatibility.  It literally broke my PC brain for the day, totally crashed h-ster.exe.

Final Specs

After painstaking review, I went with the following components. I've put down detailed decision making process below in case it may be of use to anyone else.  It doesn't mean I recommend these components but you may be interested in the why and how I chose them. (I've put amazon links not to sell anything since I am not part of  an affiliate program but just as convenience to myself.)

General Decision Factors

FUTURE PROOFING- By virtue of building a desktop,  I could continuously swap out parts for probably a decade. I wanted head space to grow and base components like the motherboard had to be upgradeable with ample DIMM slots and PCIe support for SLI. I don't run multiple GPU configuration now because current VR games sadly do not take advantage of SLI. However it could be supported in the future so I was careful to choose a mobo with at least twin SLI support.

PERFORMANCE/COST Value Ratio
I want sufficiently high quality components without burning cash like an idiot.  At the high end, you get diminishing returns at an exponential cost. I could buy faster 4000Ghz DDR4($409) instead of 3200Ghz($130 on sale) but I do plan to overclock a scootch which is probably the optimum performance gain I will see in game. For $60 more on the 4400Ghz, you can get an entire i7-9700! Since my sweet husband is financing this build, I find myself being more budget minded than I usually am.  However I did dump a hundo more on the overpriced i9-9900K because.... because reasons. This PC is for making sure I have a high VR quality gaming experience for me and my friends for years to come.

RELIABILITY- Building a custom rig, I am on my own to troubleshoot individual components to send to the manufacturer so I went with brands that appeared to be the most reliable. With next generation components, it's hard to tell what is reliable since reviewers do such short-term performance tests. I had to parse through reviews and reddit threads looking for prevalence of problems.

AESTHETICS- I am not a fan of RGB, gold, or diamond encrusted RAM- way too flashy.  However this giant box was going to be a major part of the living room visible from the dining room and kitchen so I had to go with something that I would like looking at while eating breakfast everyday.  

CPU i9-9900k vs i7-9700k

Given Intel has stagnated and resorted to pulling hyperthreading out of the i7-9700K(a feature most core i7 had for years) to differentiate the i9-9900k,  the i9-9900K touted "the fastest gaming computer in the world" might be the last Intel chip I buy.   I'm not ready to go the Ryzen way just yet, AMD gives you overall less options with regards to motherboards and components.  

The gaming benchmarks indicate the 9th gen i9-9900k performs only a few whiskers faster(if that) than i7-9700k for $100+ more and probably I would not notice a difference in game. Yes, everybody knows i9-9900k is poor value for money but still i7-9700K is 8 core 8 threads while i9-9900K is 8 core 16 threads and that hyperthreading will totally come in handy with my future ahem video processing plans!  (In my mind, this is nothing compared to macbook upcharge.)

DRAM DDR4  3Ghz+ 

When did choosing DRAM become so complicated?  Memory is something I upgraded regularly for 25+ years and I don't remember having to think this hard.  I guess there has been never such a good time to be alive with so many RAM options.
  • Capacity: 16GB.  I've been using 16GB on my alienware all this time without exceeding 12GB. With Amazon Prime next day shipping, no need to pre buy too much memory until you need it since prices will be dropping all 2019. 
  • Speed: 3200MHz DDR4 is the sweet spot of gaming performance to value at the time of purchase as the Corsair Vengeance LPX was on super sale ($130 from $219).
  • Overclocking- probably you couldn't push RAM that hard b/c the vendor would bin it for higher speed category (3333Mhz,3466Mhz)
  • Brand:  Was going to get whatever top brand was cheapest, most likely a Corsair or G.Skill
  • aesthetics: No RGB although I regret my decision. I should have used the RGB to display DRAM temperatures.
I went with  Corsair Vengeance LPX 16GB (2x8GB) DDR4 DRAM 3200MHz  ($131, now even dropping further to $122).

I also relied on gaming DRAM benchmarks to figure out the performance to cost ratio.
https://www.techspot.com/article/1171-ddr4-4000-mhz-performance/page3.html

GPU GTX 1080ti vs RTX 2080ti

Right now, with a CPU throttle, I barely break 70% usage on my 1080ti.  Given I don't play and won't play games where real time ray tracing would matter,  I don't need to shell out $1400+(with CA tax) more to upgrade for possibly ~30% boost. At that price, I should expect at least 66% improvement, no?  (You can read my very detailed post here on the 1080ti selection.)

Like many, I was truly underwhelmed with Nvidia's RTX launch- IMHO they focussed on the wrong things and let AMD beat them first to a 7nm GPU.  You know Nvidia couldn't sell ray tracing when Jensen Huang kept blathering on about "it just works"-  an infamous quote for which Todd Howard is lampooned regularly. DLSS(deep learning super sampling) is even worse and totally useless for most of the games out there- who's gonna pay to have in their game crunched through hyper expensive NVIDIA's DGX SaturnV supercomputing cluster to build a learning model for subpar results.  Games I play- heavily modded dynamic games, older titles, and indy games- will never have RTX's DLSS.  DLSS is misleading since it's upscaling 1080p for performance gain. Jensen Huang should play Red Dead Redemption 2 on PS4 Pro at 4k just to what performance magic is.

Motherboard Selection Agony

Selecting my mobo was probably my hardest decision of the entire exercise.

My basic requirements:
  • Features
    • form factor- ATX. I briefly fantasized about an eATX going dual socket but came to my senses once I chose the i9-9900k 
    • chipset: latest z390
    • SLI support (2x)
    • Reasonably powerful VRM- no doubled up phases like Asus
    • integrated WIFI
    • at least 2 m.2 slots, as many SATA ports as possible
    • RAM- 4 slot dual channel good enough
    • integrated graphics support- this wasn't originally on my list as I blithely thought all mobos supported this until I ran into one that didn't.
    • PCIe armor, i.e. fortified steel PCIe slots- these here GPUs are heavy and GPU sag on a horizontal mount is a real thing, I'm down by less than centimeter right now.
  • Reliability
  • price- The thought of spending $500 on a mobo seems wrong some how and I was shooting for midrange ~$250. 
  • pretty good overclocking performance (it didn't have to be the best given all my other requirements)

Z390 Motherboard Selection: MSI vs Gigabyte vs Asrock

For z390 boards, you pretty much have MSI, Gigabyte, Asrock. Asus had a controversy over the "fake" doubled up phases on their VRM and since Hardware Unboxed showed it being significantly hotter than the rest of the z390 boards,  Asus ROG XI Maximus Hero was not in the running.

I had planned to get one from the much touted Gigabyte Aorus line but there were so many bad Newegg reviews and a reddit thread complaining of mobo woes. The last thing you want is a temperamental motherboard. Same with ASRock Z390 Taichi Ultimate which had much better specs(3 m.2s) than my MSI selection but had some nasty Newegg reviews.  Let's just say at the time of selection, MSI while middle of the pack in terms of performance and features but had the most benign least scary complaints - 15/15 all 5 stars on NewEgg.

MSI has multiple z390 models for each of their MPG and higher end MEG line.  I had planned to get an MSI Z390 MEG ACE but in looking at the IO panel, I noticed it had no Integrated Graphics support! What if your GPU dies or won't work with your BIOS without extra configuration.

I ended up with the sensible if not boring MSI z390 Gamer pro Carbon AC (MSI, spec) after going through this exhaustive 20 page review- https://www.guru3d.com/articles-pages/msi-mpg-z390-gaming-pro-carbon-review,1.html with special attention to the overclocking page.) Every board has it's strengths and weaknesses but I ultimately chose perceived reliability.

Power Supply


I had 3 decisions forks:
  • wattage - Unlike RAM, choosing wattage is somewhat Goldilocks situation where more is not better since you can pay up the wazoo in electricity bills.  However I wanted to reserve head room for adding more power hungry components like a second GPU. The Corsair HX1250 watt was on sale for $129 as the 850 watt version and I almost got too greedy until I computed the PG&E bill.  There is a sweet spot of peak efficiency and so I wanted to be at 60%.  You can compute your wattage on-line with https://outervision.com/power-supply-calculator and mine came to 506 watts with 1 gpu (756 watts for 2gpu) so I went with a 850 watt. 
  • features - fully modular, over voltage protection, under voltage protection (but these come with most high end PSUs nowadays)
  • brand - Kind of a no brainer for me and just went with a trusty Corsair with the 10 year warranty.  I could have easily gone with the EVGA Supernova, but my brain had been sandblasted by mobo decision making.
  • model - Corsair has multiple lines, CSX/RMX/HX/AX based on efficiency 80 plus certification levels and few features. HX (Gold, fully modular,OVP,UVP) was good enough for me. I didn't see paying $50 extra for a few percentage points gain for the top AX line when I would be only running 50-70% anyway. 
(https://thepcenthusiast.com/corsair-rm-vs-hx-vs-ax-power-supply/)


Air Cooling vs Liquid Cooling

After much agonizing, I decided to stick with tried and true air cooling cuz a water cooling system will eventually break your heart.  Custom loops are not for me- too much maintenance and high high chance of leakage (see all the youtube water cooling proponents/experts JazZ2cents and bitwit crying over unexpected leaks.) Even with the all-in-one AIO units, certain small percentage leak- ample evidence on youtube and reviews. Unfortunately there is no 100% leak proof system, someone somewhere will have water sprayed all of their system through no fault of their own and that's not going to be me. Since I'm am only overclocking moderately for gaming boost, I could not see a worthwhile payoff in liquid cooling besides possibly saving space but my case is a cavern.

In the air cooling space, there is only one undisputed king- the Noctua NH-15. Noctua performs just as well if not better as some AIO Liquid Cooling solutions.  It's quieter, and gives you a 6 year warranty.   Too good to be true?

Yes, there is one catch, actually 2 (no actually 3). You have carefully curated the LED aesthetics of your case and now you have insert this weird prosthetic flesh colored thing with brown fans- obviously designed by a synth.

There is also a second catch. This thing is huuuge dominating the motherboard but I looked up compatibility charts and decided that skin colored fan frame would not deter me.

2/1/2019 Update- Due to it's bulky size and design, the fan install was the most painful part of my build. (See Part II)  Yes I could technically fit this fan on the MSI board but it forced me to pull out the power cable in front on the blue cable bar messing up the clean look. But the weird prosthetic double fans already ruined any hope of a pretty case. But all is forgiven since it keeps the CPU cool.

Case: NZXT H700 vs Phantek Evolve X

There is a dizzying selection of tower cases all with their nifty design innovations balancing airflow and sound.  Here's what I thought I was looking for in order of importance although I'm not going to lie. Phantek Evolve X is the superior case but it just didn't come in a snappy blue like the NZXT...
  • Size - Mid-tower because I'd have to get rid of the wood stove to make space for a full tower. For those using laptops, a mid-tower is a cavern. You could probably raise 4 rabbits in it.
  • Modularity and Accessibility
    • easy side panel, top panel, PSU access
  • Cooling - It didn't have to be the absolute coolest, just balanced and not a known heat trap.
  • Cable Management
  • Aesthetics
    • Not too zany with 1 glass side panel
After working with the NZXT H700 Case, I'm moderately satisfied. I'd buy it again because well,  the color of sea and sky, what's there not to love.  The blue highlight matches so well my personality since  I just don't like boring sleek grey(anthracite grey...)/black/silver tones so prevalent in all gadgetry now aways. Sadly the iconic blue cable bar is bent just too high so the motherboard 24pin cable bends very awkward underneath.

Component Vendor: Amazon vs Newegg

The only reason I went with Amazon is that due to Prime, I ordered everything on Saturday and had it shipped by Monday noon. I booted up my new machine by 5pm Monday.  Take that Digital Storm.

Weirdly, I put everything into Newegg since it appeared they had cheaper prices than Amazon. Amazon must do some tracking because when I kept going back and forth, Amazon's price had normalized to be identical to Newegg prices in my cart. However be aware that while Newegg has a price match guarantee, Amazon does NOT.

How Much Money Did I Save DIY?

I had planned to buy a build from OriginPC or Digital Storm but still use my original GTX 1080ti. OriginPC appears not to let you purchase a system without a GPU at least on-line but Digital Storm does. Plus DS is out of Fremont so I could go pick it up reducing any shipping mishaps. I had spec'ed out their $2700+ Velox when I noticed that they had a crazy long processing date "Ship Within 20-25 Business Days ".  A month or more!!! Yelp reviews confirmed such long dates and worse if you tried to cancel the order, they would charge you a 5% fee even though they hadn't started building. Combined with unacceptable wait times and an exorbitant integrator's upcharge ($2500 for a desktop without a GPU!), I realized I could easily build my own exactly how I want it for much much cheaper.

From Digital Storm, Velox $2,512.00
Chassis Model: Digital Storm Velox 
Processor: Intel Core i9-9900K  
Motherboard: MSI MPG Z390 GAMING PRO CARBON AC 
System Memory: 16GB DDR4 3000MHz Corsair Vengeance RGB Pro (RGB Light Bar)  
Power Supply: 850W Corsair RM850x (Fully Modular)   
Storage Set 1: 1x SSD M.2 (512GB Samsung 970 PRO) 
Storage Set 2: 1x Storage (2TB Seagate / Toshiba)
Extreme Cooling: H20: Stage 2: Corsair H100i PRO - 240mm Liquid CPU Cooler  

The Amazon purchase came under $1700 with better components (faster ram, better PSU) but I need to do a full spreadsheet analysis but nobody's got the time right now. Benchmarks to run. Tuning and overclocking to get on.

Wednesday, January 16, 2019

Fallout 4 VR- Wasteland Architecture Part II - Spectacle Island

By land mass, Spectacle Island is the largest buildable settlement in Fallout giving the player a lot of flexibility to experiment with more sizeable structures.  The real life Spectacle Island in Boston Harbor was named so for the early natural shape of two vaguely round land masses connected by a think strip. After decades of trash dumping and a final massive dump from the Big Dig, it looks more like a small "b" or a "9" or worse. Since Fallout 4 history diverges in 1950, the in game Spectacle Island is shaped like an amoeba.

Unlike Graygarden, I used this island as more of a blank slate since there wasn't any natively interesting structures preexisting on the island. I used my favorite cleanup mod to get rid of unscrappable garbage piles, Clean and Simple.  


Super mutants, raiders, and complaining settlers can harsh your mellow and I wanted to build a quiet seaside getaway to decompress from all the wasteland violence.  This happily has become my remote outdoor base where I do most of my crafting outdoors without being harassed by NPCs- enemy or otherwise.  Since I am cooped up at home inside a VR headset, I want to spend as much of the game time in the virtual outdoors- radiation be damned.   The first problem was unexpected- after fighting off the waves of oversized mirelurks, I walked around the island looking for possible build sites. The ocean/wind sound was so deafening I could not continue without a sound reduction mod.
.
For the main living areas, I wanted to connect railcars together and it's the fantastic Custom Concrete Walls with Window Glass mod that really tied this together. I had originally used SOE institute glass assets which didn't match as well.  The island has an uneven topology so concrete base support was used for some token amount of visual anchoring although the game engine lets you have floating detached structures. Fallout severely limits where you can snap/place items preventing me from putting supports exactly where I wanted. I also didn't want to risk corrupting my save with some potentially unstable mods so the support base is the few places the engine would let me snap.

Hall before Guest Bedroom
Most of the sides of box cars are closed off with concrete framed windows so Dogmeat wouldn't fall off.  Even if he is immortal and survives the fall, sometimes he would get stuck in circles underneath the deck and you would see a pair of twitchy ears clipping through- a total immersion killer.

The railcars are connected and placed so each window would have the most pleasing views of itself and the surrounding island. Since there is so much land here, I could have gone nuts and added dozens of railcars for a giant maze complex but it didn't feel right after about 7 railcars and I pared it back as it felt too gratuitous and unwieldy.



VR Bedtime
I keep most of the interior relatively minimal with most of it is empty except an small armory, lab, bed room, and a guest room- a huge departure from my hoarding Skyrim days.  Most of Fallout 4 exteriors and interiors are so piled with junk that one really needs visual relief in game.  My bedroom has a sleeping bag, chair, suitcase, and a doggie bed. I use the camping mod so I sleep in various places in the Commonwealth but I like to return to this home at the end of my play session.





Hung with Immersive Toilet Paper
I originally put the bathroom inside but it's impossible to get privacy in such a transparent structure and I didn't want to cover up anything.  I ended up placing the waste facilities just underneath the bedroom. Why do I even need to add a bathroom in this virtual world? Friend, if you need to ask- the answer is all about raising immersion levels.  Just look at the immersive toilet paper flipped the correct way. I don't actually use it in game and it's out of the way enough that followers will not avail themselves so it's here purely for visual immersion.  (Yes I am aware of "needs" mods but I'm not that kind of player.)  I will have to put in a magazine rack once I have enough collected.

Chemistry Workshop with Best Wasteland Views

This was the perfect corner for a chemistry station as it's visible from a window in the front entrance hall so you know you have to turn left and not get lost in the maze.  All other crafting- weapons/ammo/food is outside surrounding the original unscrappable workshop.  

Some practical Spectacle Island tips:
  • You don't need Preston Garvey/Minuteman to take over this island, just fight off waves and waves of Mirelurks and restore power to the island.
  • Custom Concrete Walls with Window Glass mod does require the Wasteland Workshop which I bought for pancake Fallout 4 and copied over into the VR folder.  




Monday, January 14, 2019

Fallout 4 VR- Wasteland Architecture Part I - Graygarden

The magic of moddable Bethesda open world games lies in the power to repurpose the base game to suit the player that diverges entirely with the main intended gameplay.   I slogged through enough of the Fallout settlement building of pitiful shanty towns to see that kind of grind was not for me. Why waste precious game time looting spatulas off ghouls for that 1 unit of steel and rubber when you can just have infinite build materials at the click of a console command.  Plus there just isn't enough Nuka Cola bottles in all of the Commonwealth to build the kind of structures I wanted.

Fallout 4 didn't have to be just a pseudo FPS/RPG.  When I saw the beauty of the fractured freeway of Graygarden, I realized Fallout 4 VR is a very unique VR construction simulator albeit sited in a nuclear wasteland.  I need not limit myself to lore friendly structures and I could go all out and build whatever a modded system would let me.  However the post apocalyptic wasteland of Fallout contains aesthetically compelling sites of broken infrastructure that I didn't want to cover over but highlight it in all it's ruined glory.


With Graygarden, I felt compelled not to use any mods to artificially clean up or restore the location.   I relied heavily on Settlement Object Expansion Pack using Institute glass assets.   Using glass floors allowed a safe navigable structure (slipping is not a thing in the Creation Engine) and still reveal the these lovingly modeled splintered rebars and dangling bits of concrete beneath. It would have been a high crime against immersion to cover these up.
 

The joy of architecting in the game world is that even though glass ramps are a dangerous slip hazard no one would build in the real world, they work wonderfully in game. Not only do they provide unique views below, these institute glass ceiling assets are wide enough that Dogmeat and other followers do not get stuck or fall off as they do on regular ladders which I had originally before I found the SOE mod.   I was never so completely engaged in VR as I did building out Graygarden and I was able to power through a year end 40 hour real life fast.  Puzzling out how to structure the stairways to the 8 storied top deck was crazy fun as you had to  dynamically building an ascending path.  I fell to my death a few times losing an hour of work when I accidentally selected and moved the item I was standing on which has turned me into an obsessive compulsive saver- but also a heavily modded Workshop wheel CTDs a lot, a lot. 

My initial plan was to build as many flat level platforms as possible for settlement building for Sim Settlements. However after building everything out, I realized I didn't want to clutter this space with repetitive immersion breaking NPC settlers. I got enough of them back in Sanctuary Hills and Taffington Boathouse. So I maintain Graygarden as a robot colony true to it's historic origins and plan to make the decks into a museum.
Adding to my immersion, I kept getting hit with heavy radiation storms during construction and trusty Codsworth kept warning me "Radiation Mum".  I could have just flipped the weather to clear but it was fun to build under adverse conditions popping all the Rad-X and Radaway I had stockpiled.  The top deck displays some custom power armor and larger weaponry but most importantly I have a vertibird helipad for the Immersive Travel mod which I use for flying to Spectacle Island.  I went a little overboard with decorative statues on top.


I also added a triple relay of elevators to get to the vertibird without having to ascend via the glass ramps. Sadly I wasted a few hours trying to fix the Bethesda floating elevator button bug trying 2 separate mods. It didn't fix it so I don't use the elevator much.


(You will have to take my solemn word that these photos do not do justice to the fun of exploring the actual site in VR. I had originally planned to make a video walkthrough of the site but I realize that my head bobs uncontrollably and since I move through teleportation, a video would be a most unpleasant jiggly experience for the viewer.)