the ip_pool[] table.
If it's zero, then there is no associated value.
+
+config->cluster_iam_master
+
+ If true, indicates that this node is the master for
+ the cluster. This has many consequences...
+
+config->cluster_iam_uptodate
+ On the slaves, this indicates if it's seen a full run
+ of sessions from the master, and thus it's safe to be
+ taking traffic.
+
+ On the master, this indicates that all slaves are
+ up to date. If any of the slaves aren't up to date,
+ this variable is false, and indicates that we should
+ shift to more rapid heartbeats to bring the slave
+ back up to date.
============================================================
Clustering: How it works.
At a high level, the various members of the cluster elect
-a master. All other machines become slaves. Slaves handle normal
-packet forwarding. Whenever a slave get a 'state changing' packet
-(i.e. tunnel setup/teardown, session setup etc) it _doesn't_ handle
-it, but instead forwards it to the master.
+a master. All other machines become slaves as soon as they hear
+a heartbeat from the master. Slaves handle normal packet forwarding.
+Whenever a slave get a 'state changing' packet (i.e. tunnel setup/teardown,
+session setup etc) it _doesn't_ handle it, but instead forwards it
+to the master.
'State changing' it defined to be "a packet that would cause
a change in either a session or tunnel structure that isn't just
updating the idle time or byte counters". In practise, this means
-also all LCP, IPCP, and L2TP control packets.
+almost all LCP, IPCP, and L2TP control packets.
The master then handles the packet normally, updating
the session/tunnel structures. The changed structures are then
had seen. The master normally than unicasts the missing packets
to the slave. If the master doesn't have the old packet any more
(i.e. it's outside the transmission window) then the master
-unicasts C_KILL to the slave asking it to die. (it should then
-restart, and catchup on state via the normal process).
+unicasts C_KILL to the slave asking it to die. (The slave should
+then restart, and catchup on state via the normal process).
+
+ If a slave goes for more than a few seconds without
+hearing from the master, it sends out a preemptive C_LASTSEEN.
+If the master still exists, this forces to the master to unicast
+the missed heartbeats. This is a work around for a temporary
+multicast problem. (i.e. if an IGMP probe is missed, the slave
+will temporarily stop seeing the multicast heartbeats. This
+work around prevents the slave from becoming master with
+horrible consequences).
Ping'ing:
All slaves send out a 'ping' once per second as a
multicast packet. This 'ping' contains the slave's ip address,
-and most importantly: The number of seconds from epoch
+and most importantly, the number of seconds from epoch
that the slave started up. (I.e. the value of time(2) at
-that the process started).
+that the process started). (This is the 'basetime').
+Obviously, this is never zero.
+ There is a special case. The master can send a single
+ping on shutdown to indicate that it is dead and that an
+immediate election should be held. This special ping is
+send from the master with a 'basetime' of zero.
Elections:
When there are no undefined tunnel or session structures, the
slave marks itself as 'up-to-date' and starts advertising routes
(if BGP is enabled).
+
+STONITH:
+
+ Currently, there is very minimal protection from split brain.
+In particular, there is no real STONITH protocol to stop two masters
+appearing in the event of a network problem.
+
+
+
+TODO:
+ Should slaves that have undefined sessions, and receive
+a packet from a non-existant session then forward it to the master??
+In normal practice, a slave with undefined session shouldn't be
+handling packets, but ...
+
+ There is far too much walking of large arrays (in the master
+specifically). Although this is mitigated somewhat by the
+cluster_high_{sess,tun}, this benefit is lost as that value gets
+closer to MAX{SESSION,TUNNEL}. There are two issues here:
+
+ * The tunnel, radius and tbf arrays should probably use a
+ mechanism like sessions, where grabbing a new one is a
+ single lookup rather than a walk.
+
+ * A list structure (simillarly rooted at [0].interesting) is
+ required to avoid having to walk tables periodically. As a
+ back-stop the code in the master which *does* walk the
+ arrays can mark any entry it processes as "interesting" to
+ ensure it gets looked at even if a bug causes it to be
+ otherwiase overlooked.
+
+ Support for more than 64k sessions per cluster. There is
+currently a 64k session limit because each session gets an id that global
+over the cluster (as opposed to local to the tunnel). Obviously, the tunnel
+id needs to be used in conjunction with the session id to index into
+the session table. But how?
+
+ I think the best way is to use something like page tables.
+for a given <tid,sid>, the appropriate session index is
+session[ tunnel[tid].page[sid>>10] + (sid & 1023) ]
+Where tunnel[].page[] is a 64 element array. As a tunnel
+fills up it's page block, it allocated a new 1024 session block
+from the session table and fills in the appropriate .page[]
+entry.
+
+ This should be a reasonable compromise between wasting memory
+(average 500 sessions per tunnel wasted) and speed. (Still a direct
+index without searching, but extra lookups required). Obviously
+the <6,10> split on the sid can be moved around to tune the size
+of the page table v the session table block size.
+
+ This unfortunately means that the tunnel structure HAS to
+be filled on the slave before any of the sessions on it can be used.
+
projects / l2tpns.git / blobdiff