VinylDNS Pre-requisites
VinylDNS has the following external requirements that need to be setup so that VinylDNS can operate. Those include:
- DNS - your DNS servers VinylDNS will interact with
- Database - the database houses all of VinylDNS information including history, records, zones, and users
- Message Queue - the message queue supports high-availability and throttling of commands to DNS backend servers
- LDAP - ldap supports both authentication as well as the source of truth for users that are managed inside the VinylDNS database
DNS
VinylDNS is not a DNS, rather it integrates with your existing DNS installations to enable DNS self-service and streamline DNS operations.
VinylDNS communicates to your DNS via:
DDNS
- DDNS is used for all record updatesAXFR
- Zone Transfers are used to load DNS records into the VinylDNS database.
VinylDNS communicates to your DNS using “connections”. A connection allows you to specify:
- The TSIG key name
- The TSIG key secret
- The server (and optionally port) to communicate to DNS with
There are 2 connections, one for DDNS and another for zone transfers. This allows you to use a different DNS server / key for zone transfers.
Connections (DDNS and Transfer) can be setup
per zone
- every zone can override the global default by specifying its own connections.global default
- assuming you are managing a primary system, you can configure default zone connections. When no zone connection is specified on a zone, the global defaults will be used.
Database
The VinylDNS database has a NoSQL
/ non-relational design to it. Instead of having a heavily normalized set of SQL tables
that surface in the system, VinylDNS relies on Repositories
where each Repository
is independent of each one another.
This allows implementers to best map each Repository
into the data-store of choice.
As Repositories
are independent, there are no “transactions” that span repositories. Each Repository
implementation
can choose to use transactions if it maps to multiple tables within itself.
There are links across repositories, for example the RecordSet.id
would be referenced in a RecordSetChangeRepository
.
The following are the repositories presently used by VinylDNS:
RecordSetRepository
- Instead of individual DNS records, VinylDNS works at theRRSet
. The unique key for RecordSet is therecord name
+record type
RecordChangeRepository
- The history of all changes to all records in VinylDNS. In general, some kind of pruning strategy should be implemented otherwise this could get quite largeZoneRepository
- DNS Zones and managing access to zonesZoneChangeRepository
- The history of all changes made to zones in VinylDNS. Zone changes can including syncs, updating ACL rules, changing zone ownership, etc.GroupRepository
- VinylDNS GroupsUserRepository
- VinylDNS Users. These users are typically created the first time the user logs into the portal. The user information will be pulled from LDAP, and inserted into the VinylDNS UserRepositoryMembershipRepository
- Holds a link from users to groupsGroupChangeRepository
- Holds changes to groups and membershipBatchChangeRepository
- VinylDNS allows users to submit multiple record changes across DNS zones at the same time within aBatch
TheBatchChangeRepository
holds the batch itself and all individual changes that executed in the batch.UserChangeRepository
- Holds changes to users. Currently only used in the portal.
Database Types
MySQL
VinylDNS has implemented MySQL for all repositories so a MySQL-only instance of VinylDNS is possible. Furthermore, there are two repositories that have only been implemented in MySQL:
- ZoneRepository
- BatchChangeRepository
Originally, the ZoneRepository
lived in DynamoDB. However, the access controls in VinylDNS made it very difficult
to use DynamoDB as the query interface is limited. A SQL interface with JOIN
s was required.
It should also be noted that all of the repositories have also been implemented in MySQL despite most currently running in DynamoDB in our VinylDNS instance.
Review the Setup MySQL Guide for more information.
AWS DynamoDB
VinylDNS has gone through several architecture evolutions. Along the way, DynamoDB was chosen as the data store due to the volume of data at Comcast. It is an excellent key-value store with extremely high performance characteristics.
VinylDNS has implemented DynamoDB for the following repositories:
- RecordSetRepository
- RecordChangeRepository
- ZoneChangeRepository
- GroupRepository
- UserRepository
- MembershipRepository
- GroupChangeRepository
- UserChangeRepository
Currently using DynamoDB would also require the user to either use MySQL for the batch change and zone repositories or also provide an implementation for those repositories in a different data store.
Review the Setup AWS DynamoDB Guide for more information.
Message Queues
Most operations that take place in VinylDNS use a message queue. These operations require high-availability, fault-tolerance with retry, and throttling. The message queue supports these characteristics in VinylDNS.
Some operations do not use the message queue, these include user and group changes as they do not carry the same fault-tolerance and throttling requirements.
Message Queue Types
AWS SQS
Our VinylDNS instance uses AWS SQS to fulfill its message queue service needs. SQS has the following characteristics:
- High-Availability
- Retry - in the event that a message cannot be processed, or if a node fails midstream processing, it will be automatically made available for another node to process
- Back-pressure - SQS is a pull based system, meaning that if VinylDNS is currently busy, new messages will not be pulled for processing. As soon as a node becomes available, the message will be pulled. This is much preferable to a push based system, where bottlenecks in processing could cause an increase in heap pressure in the API nodes themselves.
- Price - SQS is very reasonably priced. Comcast operates multiple message queues for different environments (dev, staging, prod, etc). The price to use SQS is in the single digit dollars per month. VinylDNS can be tuned to run exclusively in the free tier.
Review the Setup AWS SQS Guide for more information.
MySQL
VinylDNS has also implemented a message queue using MySQL, which incorporates the features that we currently utilize through AWS SQS such as changing visibility timeout and re-queuing operations.
Review the Setup MySQL Guide for more information.
LDAP
VinylDNS uses LDAP in order to authenticate users in the Portal. LDAP is not used in the API, instead the API uses its own user and group database for authentication.
When a user first logs into VinylDNS, their user information (first name, last name, user name, email) will be pulled from
LDAP, and stored in the UserRepository
. Credentials will also be generated for the user and stored encrypted in the UserRepository
.
Review the Setup LDAP Guide for more information