IT Infrastructure | Virtualization | Application Delivery | Networking | Storage


Leave a comment

How to install the Kaseya VSA Agent on a non-persistent machine

Deploying Kaseya VSA agents to provisioned workstations and servers from a standard image can be a challenge. Using the typical methods result in duplicate GUID’s due to the nature of the technology making it impossible to accurately monitor and report on these types of machines.

The following procedure has been tried a tested (successfully) numerous times on Citrix VDI/SBC environments leveraging Provisioning Server and Machine Creation Services. That being said, the process should work on other systems that are non-persistent providing they have a persistent volume where the application files can be placed.

The term ‘Target Devices’ used throughout this process is what provisioned desktops and servers are called in Citrix Provisioning server. If you are not familiar with Citrix Provisioning server, think of these as being your non-persistent machines that the image is being delivered to.

  1. The first thing we want to do is create an Organisational Unit (OU) where your update / maintenance target device(s) will live.
  2. Apply a GPO to the maintenance Organisational Unit (OU) which will stop Kaseya communicating with the master server.
  3. Create a new agent package configured to install the agent on the persistent drive, typically, the ‘writecache’.
  4. Create a new version and open the master image / vDisk in maintenance mode (read/write)
  5. Manually install the package onto the new version making sure the GPO is working which prevents communication with Kaseya (Stopping the agent from connecting to the Kaseya master server which in turn updates the KaseyaD.ini file)
  6. Once installed, go into the install location on the persistent drive and scoop out a copy of the folder labelled with the agent ID, this is typically found inside ‘Program Files (x86)\Kaseya’ (You will need this for later)
  7. Stop the Kaseya services.
  8. Delete the following 3 registry keys:
    Kaseya-Agent-PVS
  9. Perform typical cleanup tasks and shutdown the machine.
  10. Promote new version to Test/Production.
  11. Place the folder that was copied in step 3 into the persistent drive on all target devices that are provisioned from the master image / vDisk (This should be placed in the directory Kaseya expects to find it as configured in step 1 (Regedit can be used to confirm this)
  12. Reboot your target devices.

That’s it… you’re done. Provisioned / non-persistent machines should now report into the Kaseya master server with a unique GUID as do standard machines.

If Kaseya services fail to start on target devices it’s likely due to the application files being in the wrong path on the persistent drive. Confirm the application files are where the should be according to the registry.

Very keen to hear how you go. Feedback welcome, thanks

 

 

 


Leave a comment

Citrix PVS Image Process – Best Practices

On completion of the implementation and handover of any technology to an IT department / client, high quality build and configuration documentation is a must. This may seem obvious but I’ve been in many situations where project time gets squeezed due to budget or to get a deal over the line and a ‘day or two’ gets lobbed off the documentation phase. This leads to inadequate and poor quality documentation which may not be an issue at first, but down the line when someone else has to take over the environment or troubleshoot it can be nothing short of a nightmare.

If documentation is up to scratch and training workshops are provided, it will empower the client to take over and confidently manage the environment, and in some cases, carry out future upgrades. Some integrators may see this as bad business from a revenue perspective, but I believe customers are likely to consume more services in the future when they are confident you are providing them with the correct traning and tools. And, realistically, internal IT departments are typically way too busy to take on more than what they already have on their plates, but it’s good practice to give them the option.

Now back to point of this blog post; one of our clients had the time and resource to invest in the internal IT team following the positive start in documentation and training provided by us. They asked us to provide some best practice tips around the PVS imaging process in preparation for a PVS upgrade they were taking on internally. We, the Citrix Engineering and Consulting team, came up with the following, it was written over a year ago and has had the input and review of around 6 senior Citrix engineers and consultants. I still refer to it and find it relevant so here it is:

Provisioning Services: Image Creation

Assumes that the image is working with an attached local drive. The attached local drive is typically created/placed on high speed or near high speed storage according to environment needs or constrains.

The following items are configured within the image and surrounding infrastructure to optimise performance, utilisation, compatibility and management;

1. Redirect PageFile from %SystemDrive% to a local drive.

2. Crash dumps;

  • Totally disable crash dump processing,
  • Or, configure as mini dump only and redirect to the local drive.

3. Disable or minimise as much write operations to the “%SystemDrive%” as possible. Ideally, write operations should be restricted to the ““%SystemDrive%\Users” path only.
As a simple example, exclude *ALL* optimisation and scanning tools such as;

  • Disable scheduled defragmentation tasks of the “%SystemDrive%”.
  • DDL optimisations, including Citrix DDL optimisation.
  • Deploy Antivirus exclusions for key operating components and files (refer to current Citrix and vendor recommendations).

4. Windows Event logs redirected from “%SystemRoot%\System32\winevt\Logs” to the local drive. This may be done as either;

  • Policy based redirection on a per log basis,
  • Or (recommended), using junction point redirection on the “Logs” folder to capture all logs.5. Application log redirection – if required. This may be done via vendor recommended configuration or junction point redirection.

6. Keep the design simple. Complex designs introduce complex problems that are often extremely difficult to diagnose or isolate. In general, always try and use vendor default paths, settings and configuration. Only deviate from this where a vendor has a clearly defined best practice procedure to allow this.

  • It is an unfortunate fact of life that programmers often accidentally hard code configuration options into applications and the operating system. Tricky or complex deviations from vendor recommendations often lead to obscure, intermittent or strange behaviours that can be extremely difficult to diagnose.
    If you use only vendor recommended configuration changes, then you have a much higher assurance that the vendor will include full testing in their hotfixes and patching procedures.

7. All applications should be validated to confirm that they do not place ephemeral (temp, batch or data), or user specific files in in any of the following paths; %SystemDrive%, %SystemDrive%\TEMP, %SystemRoot%, %ProgramFiles%, %ProgramFiles(x86)%, %CommonProgramFiles%, %CommonProgramFiles(x86)%, or any other similar path. Note; application global configuration files used by all users are exempt from this rule.

8. All applications should be validated to confirm that they do not create server specific configuration settings. This includes both registry and file based configuration – including INI, XML, CFG, DAT, etc, files.
Note: Special attention should be made to look for server unique settings such as GUID strings like “{BB2E617C-0920-11d1-9A0B-00C04FC2D6C1}”, that may uniquely identify a server for AntiVirus, Firewall, licencing’s, etc, configuration.
If application unique settings are encountered, these unique server settings should be set via the Provisioning Services device TARGET personality strings. Thus, can be configured within the Image during each server’s boot sequence.

9. Do NOT redirect application installation paths. This may include changing %ProgramFiles%, %ProgramFiles(x86)% locations or the installation path of applications.

  • It is an unfortunate fact of life that programmers can accidentally hard code program paths into application operate. This may lead to obscure, intermittent or strange application behaviours if an application is not installed in its default path.
    In addition, be extremely careful with changing application configuration settings outside of those provided in the application’s own configuration files or registry settings.

10. Good practice application tips also include;

  • Try and minimise any situation where application executables are run from network shares. In addition, this also includes opening database files from network shares.
    ▫ It is understood that occasionally the only option is to place some applications on network shares due to limitations on their design or age.
  • Removal of any unnecessary application based Taskbar items or background services.
  • Removal of any unnecessary Start Menu or Desktop icons. This includes shell based namespace items held in the registry.

11. Utilise user “home folder” shares to store application per user settings.

12. User based Folder Redirection policy applied via GPO.

13. Deploy Citrix Profile Manager based profile management. This will ensure that multi-session &/or disconnected sessions do not encounter the synchronisation issues.

14. Consider segmenting PVS streaming traffic to a dedicated VLAN if the network has a highly utilized or potentially saturated. This can also be implemented for security purposes if the business require it. Note, segmenting PVS traffic used to be a Citrix best practice but is no longer the case for the vast majority of deployments.

15. TCP Large Send Offload option allows the AIX TCP layer to build a TCP message up to 64 KB long and send it in one call down the stack through IP and the Ethernet device driver. The adapter then re-segments the message into multiple TCP frames to transmit on the wire. The TCP packets sent on the wire are either 1500-byte frames for a Maximum Transmission Unit (MTU) of 1500 or up to 9000-byte frames for a MTU of 9000 (jumbo frames).

16. Re-segmenting and storing packets to send in large frames causes latency and timeouts to the Provisioning Server. This should be disabled on all Provisioning Servers and clients.

17. Disable Large Send Offload, open the Network Interface Card (NIC) properties and select the Advanced tab.
Some NICs do not offer this setting in the properties page. In this case, you must change a registry key change to disable Large Send Offload. To disable, add the following entry to the registry:

  • Target Device
    Note: For Provisioning Server 6.0 and beyond, BNNS driver is no longer used for Windows 7 and 2008, so this registry key is not applicable. However, BNNS is still used for windows XP and 2003.HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BNNS\Parameters\
    DWORD = EnableOffload
    Value: “0”
  • Provisioning Server and Target Device
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TCPIP\Parameters\
    Key: “DisableTaskOffload” (dword)
    Value: “1”

18. Hard code all PVS ports on the servers & clients to avoid auto negotiation network speed, this can cause slow start up times and PXE timeouts for target devices.

19. Firewall and server to server communication ports.

Open the following ports in both directions:

  • UDP 6892 and 6904 (For Soap to Soap communication – MAPI and IPC)
  • UDP 6905 (For Soap to Stream Process Manager communication)
  • UDP 6894 (For Soap to Stream Service communication)
  • UDP 6898 (For Soap to Mgmt Daemon communication)
  • UDP 6895 (For Inventory to Inventory communication)
  • UDP 6903 (For Notifier to Notifier Communication)

Provisioning Services: Image Closure

The provisioned image should typically assume a local drive is attached for the storage of; WriteCache, PageFile redirection, Windows Mini dumps (only if absolutely required), Windows Event Logs, and Application specific logs.

Any other server specific personality settings should be set via the Provisioning Services device TARGET personality strings.
Confirm the following:

1. Refer to the “Image Creation” sequence and validate all items. Specific focus should be on any additional applications that have been added or modified in the environment. Then focus on the best practice steps associated with applications. E.g.

  • Personality strings set, as required, for applications that require personality traits.
  • Confirmation of shared folder usage. Specifically; applications should not place ephemeral (temp, batch or data), or user specific files in %ProgramFiles%, %ProgramFiles(x86)%, %CommonProgramFiles%, %CommonProgramFiles(x86)%, %SystemDrive%, or %WinDir% paths.

2. Using a Domain account with local Admin rights, login and reboot the server. If you wish, you may also perform a final “GPUPDATE /FORCE”.

3. Login as the local machine Administrator account and remove:

  • Any roaming or local user profiles on the server image created or left from the creation or update of the server image. This must be validated in three (3) locations;
    ▫ Remove the extraneous profiles using the Windows Profiles dialog under the “Advance System Settings”.
    ▫ Verify there are no extraneous profile folders in “%SystemDrive%\Users”.
    ▫ Using regedit, verify there are no extraneous profiles in “HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList”. Specific attention should be focused on any “.bak” files.
  • Any / all temp files created on the %SystemDrive% &/or sub folders.

4. Using the Citrix Role Manager, perform a XenApp prep for image closure. Do not remove the server from the Farm. Then shutdown gracefully in preparation for image closure in PVS.

5. When using Citrix UPM 4.x in the image, remove the UPM cache file (not applicable to UPM 5.x)

6. Run ‘ngen /ExecuteQueuedItems’ for all CLRs (2.0 x86, 2.0 x64, 4.0 x86, 4.0 x64) in order to precompile assemblies, if this step is not performed, ngen will run the pre-compilation at random times after every reboot in PVS standard mode and therefore cause significant IO load.

Thanks for reading, share if you found this useful.


Leave a comment

Securing Citrix Netscaler… The basics!

While there is an abundance of best practices and white papers on how to secure your Netscaler, I come across many implementations that are worryingly insecure. Whenever I highlight this with the IT Manager, engineering or the security team they are naturally keen to plug these holes asap.

After some digging I normally find it’s due to lack of understanding in the product, a disjoint in the handover from the integrator (if installed by a 3rd party), or the project budget was running out and corners were cut. Maybe it went in as a POC and somehow slipped into production as it was ‘working’. It’s particularly prevalent in businesses where there is an absence of a dedicated network / security team and the senior ‘all rounder’ techs manage the routing, switching and firewalls but aren’t too sure of the mystical box called Netscaler that someone installed at some point.

Regardless of the reasons, it must be a least, somewhat secured!

Netscaler is an awesome but complex product, unlike it’s main competitor, the F5 which is modular, the Netscaler is an all in one, unified system, you just need to license the appliance for the features you need. That also explains why the answers vary when you ask, “What is a Netscaler?”

“It’s a secure access gateway”, “it’s a load balancer”, “it’s an application accelerator”, “it’s an SSL offloader”… Well, it’s all of those things and a whole lot more…

Here are 10 tips I’ve thrown together that will minimize the attack surface and provide some security for your Netscaler implementation. I recommended further securing the Netscaler as per Citrix best practice but these steps will at least get it somewhat secured in around an hour.

1.Change the default login! Yes, user: nsroot password: nsroot is left in place way too often.

2. If running a physical appliance (MPX), ensure it’s physically secured in a comms room with limited access to the front panel & console port.

3. Configure role-based access security control (RBAC) for the admins and engineers that require access to the device with named accounts for each.

4. Configure a low system session timeout for the GUI and CLI. This can be done at user / group level but before going that granular, it can be set globally:

  • GUI: Navigate to System > Settings, click Set global system parameters, and set the ANY Client Idle Time-out (secs) parameter.
  • CLI: At the command prompt, enter the following command:
    set system parameter -timeout <secs>

5. Use HTTPS for GUI management access, disable the HTTP access to the GUI management interface. To do so, run the following command:

  • > set ns ip <NSIP> -gui SECUREONLY

6. Create a 2048-bit RSA private and public key pair and use the keys for HTTPS and SSH to access NetScaler IP address, instead of using the factory provisioned 512-bit RSA private and public key pair.

7. Patch it! Ensure the latest security patches and known stable firmware are applied.

8. Ensure it’s secured by a firewall and that it’s management IP is not accessible from the internet.

9. Configure logging to an external host, there’s a nice walk through here:

10. Use Access Control Lists (ACLs) so that the Netscaler CLI and GUI is only accessible from controlled management VLANs / network segments.

I must stress, you can go much further in securing the Netscaler but the above points are fairly easy to implement and will provide a nice baseline. It should bring some value to those sitting with a wide open, unsecure appliance, and believe me, there’s plenty of them!

Feedback and ideas are welcomed, as always! thanks

 


Leave a comment

SD-WAN with Citrix Cloudbridge

SDN is a technology that I’m interested in but I’ve yet to get my hands dirty, being a ‘Citrix Guy’, I’ve been looking at their SD-WAN solution using Cloudbridge. I wasn’t fortunate enough to make this years Citrix summit in Las Vegas but I enjoyed this demo of it in action, albeit a ‘wee’ bit cheesy…  check it out here:

 


Leave a comment

How to add additional local storage to Xenserver 6.5 post install

At the tail end of last year I decided to run up a lab for testing Machine Creation Services using Xenserver 6.5 in preparation for a project I had in the pipeline. My lab is currently running vSphere 6 so I decided to nest a Xenserver 6.5 host inside the EXS host.

I quickly found that I required more local storage on the nested host to support my testing…

Here’s how to do it…

Select the Nested Host (VM) within vSphere / vCenter:

XS1

Edit the Virtual Machine and add an additional vDisk, in this case, I’ve added a 150GB as shown below:

XS2

Once the vDisk is added, reboot the VM (Nested Xenserver Host) then login to the Local Commad Shell:

XS3

By running ‘fdisk -l’ you will see all disks accessible by the host including the recently added, uninitialized volume.

Currently active (initialized) disks (sda, sdb & sdc):

XS4

Newly added disk (sdd):

XS5

NOTE, If this is the first drive you have added, it will likely enumerate as ‘sdb’ (Storage Device B). This is the 4th additional local disk I have added. I’ve still to confirm but I reckon that’s why I’ve got ‘sdd'(Storage Device D):

Now that the new disk is visible to Xenserver, it must be initialized so that can be used by Logical Volume Manager (LVM). Do this by running:

‘pvcreate /dev/sdd’

XS6

Remember! Your newly added volume will likely be ‘sdb’ opposed to ‘sdd’!

Once the disk has been initialized, we need to create a Xenserver Storage Repository so it can be used by the Xenserver Hypervisor:

xe sr-create type=lvm content-type=user device-config:device=/dev/sdb name-label=LS4′ 

XS7

I have chosen to label it ‘LS4’ (Local Storage 4) as it’s the fourth disk I have added.

The new, additional local volume should be visible within Xencenter within seconds:

XS8

The above process would work in a physical Xenserver host if more local disk was required but I reckon that would be an unlikely scenario… but who knows in this game…