Thursday, May 7, 2009

Designing and Developing a Debugger for a Language over TCP/IP and HTTP

Developing a Debugger over TCP/IP and HTTP


The goal is to understand how to create a debugger framework for any domain model. Here we create a debugger for an imaginary XML-based language say SML (Service Markup Language)

We won't go into the details of Language Semantics and what domain model it represents.
Also we shall assume there is a language interpreter and AST-Generator which will return the list of variables and there processed values upon receiving a line number from breakpoint debug event.

Sample code to debug :

L1 :
L2:
L3:
L4:

L5:

Architecture of Service Markup Language Debugger







1.Debug Launcher will start the Debugger and establish a communication (TCP / HTTP) with the (Local / Remote) Debug Target.
2.The Debug View will be rendered and every Debug Event (STEP_INTO, STEP_OVER, RESUME) will be trapped by the DebugEventListner.
3.Event Listener will submit the SML Service Processing request via a DebuggerProxy to the local/server host.
4.Finally, DebugEventProcessor with the help of Language Interpreter, AST Utilities and Service Repo will generate the Debug Response
5. and send it back to the Listening Debugger Client (TCP Client Socket / Debugger Web Client).
6.The Debugger Client in turn will notify the DebugEventListner
7.Event Listener will refresh the Debug View to show variables and values an will also print in the console and do whatever else needed



The sample code does not provide the complete implementation of the debugger flow. It gives good idea
about how to create a debugger over TCP or Http


The following pictures depicts the above mentioned flow





How to implement the above mentioned Debug-Flow ?

A. Launching Debugger : com.pramati.sml.debugger.launcher
B. Client-Side Debug Event Handling : com.pramati.sml.debugger.core, com.pramati.sml.debugger.client
C. Server-Side Debug Event Hanling : com.pramati.sml.debugger.server


A. Launching Debugger :

1.Define Launch Configuration Type and Launch Delegate
- Contribute any
- Specify delegate if needed (upgrade resource mappings)
- Implement ILaunchConfiugrationDelegate
- Contribute with or use for existing types

2.Associate the Launch Delegate with Launch Tabs
- Contribute
- create SMLDebuggerLaunchTabGroup to show SMLDebuggerIntegrationTab
- create SMLDebuggerIntegrationTab extends AbstractLaunchConfigurationTab
implements IPreferenceChangeListener
** The list of servers actually managed in preference page
** The list is displayed in the combo box of launch configuration page
maintain resource mappings for contextual launch
associate SMLDebuggerIntegrationTab with configurationType

3.SMLDebuggerLaunchConfigurationDelegate will either invoke DebuggerOnTCP or DebuggerOnWeb both of which extends StandardVMDebugger.

SMLDebuggerLaunchConfigurationDelegate will create a new VM instance and capture the Launch-Configuration details from the Dialog Page.






>> DebuggerProxy will notify SMLClientEventHandler once it receives any Server response. DebuggerProxy will send the DebugEvent requests.
>> SMLClientEventHandler will accordingly handle the Debug Events and delegate the tasks of refreshing/updating the View to DebugTarget
>> SMLClientEventHandler will be invoked if any UI event (resume/step over/suspend) takes place and accordingly it will submit the request to the DebuggerProxy who will in turn post it to the server.



` B. Client-Side Debug-Event Handling

  




DebugEvent

Controller



VIEW CONTROLLER MODEL


After defining a launch configuration, we should create
Debug VIEW :
(1) make source code editor adapt to breakpoints and show markers,
> Figure out what types of breakpoints are needed and accordingly we have to contribute
an extension.
> Define extension for each breakpoint
> Install the breakpoints in the debug target through IbreakpointListener interface.
> Implement BreakpointManagerListener in your debug target to support “skip breakpoints”.
> Create an implementation of an IToggleBreakpointsTarget adapter using, and register the adapter with your editor
> Contribute a RulerToggleBreakpointActionDelegate to your editor using the
extension point

(2)provide presentation UI (label, image) for breakpoints, variables
> Contribute a extension
> Provide corresponding implementation of IDebugModelPresentation , which is an ILabelProvider
> Whenever a breakpoint attribute changes the breakpoint label and image are modified

(3)and provide sourcePathComputers for sychronizing stack frame-threads with source code editor.
> when a stackframe is selected

(core) >> SourcePathComputer and SourceLocator
Get the SMLSourceLookupParticipant from SMLStackFrame
>> contribute proper SMLLineBreakpoint
(ui) SMLDebugModelPresentation
provide SMLBreakpointAdapterFactory so that Editor adapts to SMLLineBreakpointAdapter - create breakpoint

Debug Model:
Implement debug elements: IDebugTarget , IThread , IStackFrame …
Implement supported capabilities: IStep , ITerminate …
Ensure model fires required DebugEvents
debug model contains
Debug Target SMLDebugTarget -> IDebugTarget
Threads SMLThread -> IThread
Stack Frames SMLStackFrame -> IStackFrame
Variables SMLVariable -> IVariable
Register Groups SMLRegisterGroup -> IRegisterGroup
Register SMLRegister
Value SMLValue


Debug Controller :
>> Events are sent over a separate socket
>> listens for new debug events
>> Either DebugUI will generate the Events or DebuggerProxy will propagate the events from Server-Side to the controller


C.Server-Side Debug-Event Handling



 Debugger in Action – whats happening behind the scene

Enough of theory ! Lets dive into sample code flow …

Debugging the sample script of ServiceMarkupLanguage









Scenario 1 : Debugging over TCP

A -- Start the Debugger Client Socket and Server Socket

The very first thing to do is – getting our launch dialog ready !
Section-A (Launching Debugger) already explains how to configure the Launcher.
SMLDebuggerLaunchTabGroup (is a AbstractLaunchConfigurationTabGroup ) – creates the tabs as per Debugger UI requirements.
SMLDebuggerIntegrationTab (is a AbstractLaunchConfigurationTab) shows the list of servers that user can configure from SMLPreferencePage.

User selects local server mode so that a SML Script will be Debugged through a new OS process in a separate JVM instance.

Hitting – ‘Launch’ will trigger SMLDebuggerLaunchConfigurationDelegate.

*** SMLDebuggerLaunchConfigurationDelegate#launchServer(ILaunchConfiguration
configuration,ILaunch launch, IProgressMonitor monitor, String mode)

LaunchDelegate will collect the LaunchConfiguration data (selected file, debugger port) from the Dialog.

Once the user is authenticated, required jars (for server-side event handling and language compiling) will be loaded on classpath and all VM configuration details (classpath, environment variables, program arguments) will be set up required for the VM.

SMLDebuggerLaunchConfigurationDelegate will pass on the VM instance to DebuggerOnTCP ( StandardVMDebugger).

*** DebuggerOnTCP#run(VMRunnerConfiguration config, ILaunch launch,
IProgressMonitor monitor)
(1) DebuggerOnTCP will spawn a new o/s process to run DebuggerHost (mainClass set up in the env) in that process.
Process p = exec(cmdLine, workingDir, envp);
IProcess process= DebugPlugin.newProcess(launch, p, label, attributes);

*** DebuggerHost will start the ServerSocket for listning to the incoming events and sending outgoing events.

ServerSocket serverReq = new ServerSocket();
serverReq.bind(new InetSocketAddress(recvReqPort));
logger.warn("Server listening for connections on ports "
+ recvReqPort);
while (true) {
final Socket session = serverReq.accept();
new Thread("Debug Session Handler") {
public void run() {
new SMLClientEventHandler(session);
}
}.start();
}

*** SMLClientEventHandler adds itself as a listener for the DEBUG_EVENTS generated by DebugEventProcessor. DebugEventProcessor actually invokes the interpreter to process the Script based on the line numbers retrieved from breakpoint markers and then create Value objects holding {variable,value} pairs. It will generate the StackFrame event which will be sent over TCP/IP by the SMLClientEventHandler.

*** It reads REQUESTs and posts RESPONSE over the same synchronized stream. (created per debugger client session)

(2) DebuggerOnTCP will connect to the target VM instance via com.sun.jdi.connect.ListeningConnector .. (Listens for one or more connections initiated by target VMs). Then it will run the DebuggerProxy which will be able to receive data from ServerSocket.

*** "com.sun.jdi.SocketListen"  Bootstrap.virtualMachineManager().listeningConnectors()

ConnectorRunnable runnable = new ConnectorRunnable(connector, map);
Thread connectThread = new Thread(runnable, "Listening Connector"); //$NON-NLS-1$
connectThread.setDaemon(true);
connectThread.start();

while (connectThread.isAlive()) {
DebuggerOnTCP will create the SMLDebugTarget (SMLClientEventHandler) which in turn
will start the SMLClientEventHandler thread.

****
SMLClientEventHandler#IStatus run(IProgressMonitor monitor) {
*** initializeDebuggerClient()
// SMLClientEventHandler will invoke the DebuggerProxy to start the Client
Socket.

while (!isTerminated() {
debuggerProxy = new DebuggerProxy();

*** requestSocket = SocketUtils.connectToLocalServer(sendReqPort);
SocketUtils.addShutDownHook(requestSocket);
new Thread() {
public void run() {
listenToServer();
}
}.start();
reqSocketInput = requestSocket.getInputStream();
reqSocketOutput = requestSocket.getOutputStream();
***
}
}
}

So, this is how the – debugger will - Send request in one thread on client side and and Receive response in another thread on TCP server side.

Once DebuggerProxy is setup, it sends the first DEBUG_EVENT request LOGIN to the DebuggerServer (DebuggerHost.java)

debugger.login
>> new ObjectOutputStream(reqSocketOutput).writeObject(request);
reqSocketOutput.flush();

SMLClientEventHandler will get the object thru the server socket ..
*** listenForRequests() {
while (!requestSocket.isClosed()) {
// check if we can receive requests.
SocketUtils.checkRecvChannel(requestSocket);
// schedule the request for running
// and sending acknowledgment to client.
synchronized (requestSocket) {
if (reqSocketInput.available() > 0) {
request = (Request) new ObjectInputStream(reqSocketInput).readObject();
if (request != null) {
String methodName = request.getDebuggerMethod();
Object[] args = request.getArgs();
if(methodName.equals("login")){
// store token info in session ..

debuggerProxy.registerForDebug()
>> ServerDebugEventProcessor stores the SML script for processing.



B – Show the Debugger UI

Event Notification is performed in a separate thread which queues up the events and fire them

EventListner listens to the debug events and then accordingly refreshes the UI … i.e. if the Event is IDebugtarget.create then expands the target, resume – select target, update label and refresh the children. Similarly for IThread.suspend – updates thread labels and top frame labels.

ClientDebugEventDelegate#started() …

1.After initializing client socket, ClientDebugEventDelegate (SMLDebugElement)shows the Debug UI
*** fires creation event : fireEvent(new DebugEvent(this, DebugEvent.CREATE));


2.Next it reads all the installed breakpoints and notifies the SMLClientEventHandler
*** fireBreakPointEvents() {
IBreakpoint[] breakpoints = DebugPlugin.getDefault().getBreakpointManager().getBreakpoints(SMLDebugConstants.SML_DEBUG_MODEL_ID);
for (int i = 0; i < breakpoints.length; i++) { clientSMLClientEventHandler#breakpointAdded(breakpoints[i]); } } *** breakpointAdded(IBreakpoint breakpoint) { if (supportsBreakpoint(breakpoint)) { boolean isManagerEnabled = DebugPlugin.getDefault().getBreakpointManager().isEnabled(); if (breakpoint.isEnabled() && isManagerEnabled) { if(clientLookupThread == null) return; RemoteDebugger client = clientLookupThread.getClient(); if(client != null){ synchronized (client) { clientLookupThread.setBreakpoint(breakpoint); } } *** ILineBreakpoint lBp = (ILineBreakpoint) breakpoint; IMarker marker = lBp.getMarker(); String scriptId = marker.getResource().getFullPath() .toPortableString(); int lineNumber = lBp.getLineNumber(); markerIdVsBreakPointLineNum.put(marker.getId(),new Integer(lineNumber)); // keep track of the Breakpoints on the Server debugger.setBreakpoint(scriptId, lineNumber, true); *** The same breakpoint information is stored on the server side scriptVsbreak points Map by ServerBreakPointManager….. SMLServerEventHandler#setBreakpoint(String scriptId, int location, boolean active) { Breakpoint breakpoint = new Breakpoint(scriptId, null, location, active); breakpointManager.setBreakpoint(breakpoint); } ServerBreakPointManager#setBreakpoint(Breakpoint breakpoint) { Map scriptBreakpoints = getScriptsBreakpoint(breakpoint.getScriptId()); scriptBreakpoints.put(breakpoint.getLocation(), breakpoint); } *** This is how all the break points get stored on the server side. DebuggerProxy receives a START Event from Server once breakpoints are stored and notifies the client-side debug event listners. SMLClientEventHandler#handleEvent(DebugEvent debugEvent) case DebugEvent.THREAD_START: SMLThread mt = new SMLThread(smlDebugTarget, threadID, debugEvent.getScriptId()); smlDebugTarget.addThread(mt); smlDebugTarget.fireCreationEvent(); **** RESUME all the threads resume() throws DebugException { // resume all threads for (int i = 0; i < fThreads.size(); i++) { IThread indexThread = (IThread) fThreads.get(i); indexThread.resume(); **** SMLClientEventHandler#resume() – first sends the script Id corresponding to this thread to the server so that any request to process an ASTStatement at a particular line number can be performed properly. *** String scriptId = smlThread.getScriptId(); debugger.setStepMode(scriptId, false); String modelIdentifier = smlThread.getModelIdentifier(); ILaunchConfiguration launchConfiguration = getLaunchConfiguration(); HashMap inputParameters = DebugUtils.getInputParameters(launchConfiguration); debugger.resume(scriptId, modelIdentifier, inputParameters); **** ServerDebugEventProcessor – sets the scriptId as the current scripted of the Interpreter Scrip-Execution Context and delegates the execution request to Interpreter. Interpreter extracts the ASTStatement with given line number (provided thru the params) and executes it. The execution logic populates locationVsVariable map. The Command notifies the listeners i.e. SMLClientEventHandler. **** if ( breakpointManager.isStepMode(scriptId) ) context.getEventManager().broadcastListeners(scriptId, DebugEvent.STEP_OVER, currentLineNumber); else if ( breakpointManager.hasBreakpoint(scriptId, currentLineNumber) && breakpointManager.isBreakPointActive(scriptId, currentLineNumber) ) context.getEventManager().broadcastListeners(scriptId, DebugEvent.BREAKPOINT, currentLineNumber); } SMLClientEventHandler – retrieves variable name against location and sets proper value to the variable. **** suspends server-side threads int location = debugEvent.getLocation(); if (debugEvent.getEventId() == DebugEvent.BREAKPOINT) { Thread currentThread = Thread.currentThread(); tm.suspendThread(currentThread,location); }else if(debugEvent.getEventId() == DebugEvent.STEP_OVER) { String scriptId = debugEvent.getScriptId(); boolean stepMode = debuggee.getBreakpointManager().isStepMode(scriptId); if(stepMode){ Thread currentThread = Thread.currentThread(); tm.suspendThread(currentThread,location); } } SMLClientEventHandler – posts the DebugEvent Response SMLClientEventHandler#handleEvent(..) .. receives BreakPoint notification : It sends STACK_FRAME event to server to get the values for the variables. At the same time, it notifies SMLClientEventHandler to suspend the SMLThreads DebugTargetProxy in turn handles the events and notifies ModelChnageListners to refresh the views (fireModelChanged (IModelDelta delta)…) case DebugEvent.BREAKPOINT: { // if a break point is hit request for all the variables debugger.getStackFrame(threadID); smlDebugTarget.suspend(); Show the StackFrame : client receives STACKFRAME Event >> create SMLStackFrame

case DebugEvent.STACKFRAME:
Debugdata rd = (Debugdata) data;
SMLThread mt = smlDebugTarget.getThread(threadID);
String scriptId = mt.getScriptId();
IResource resource = getResource(scriptId);
if(resource != null){
SMLStackFrame sf = new SMLStackFrame(mt, resource.getName(),
debugEvent.getLocation(), rd);

ArrayList stackFrames = getStackFramesForThread(threadID);
stackFrames.clear();
stackFrames.add(sf);
// viewed stack frame.


now show the data .. Debug Flow has to stop ..
>> getThread(threadID).suspend();

smlDebugTarget.suspended(threadID,org.eclipse.debug.core.DebugEvent.BREAKPOINT);

highlights the current breakpoint in the source code.

Scenario-2 : Debugging over the web

runner = new RemoteSMLServerDebugger(vm, launch); // debug mode for remote

private void submitRequest(Request request) {
Response response = transport.publishRequestAndGetResponse(request);
if (response.getException() != null) {
Throwable ex = response.getException();
if(ex instanceof JEMSException){
throw (JEMSException)ex;
}else{
throw new RuntimeException(response.getException());
}
}
DebugEvent[] events = response.getEvents();
if (events != null) {
for (int i = 0; i < events.length; i++) {
fireDebugEvent(events[i]);
}
}

}

The Rest is same as debugging over tcp.

Now so far we have seen how communication is established between debug client an debug target (local / remote) and debug events are processed both at the Client and Server side.

Next few obvious questions pop up :

1. How does break point get created upon toggle / double click ?

Toggle breakpoint Action asks the active workbench part () for its its IToggleBreakpointsTarget adapter. The action interacts with the adapter class SMLLineBreakpointAdapter#toggleLineBreakpoints(IWorkbenchPart part, ISelection selection)
The editor gets a call back for decorating.
SMLDebugModelPresentation provides labels and images for breakpoint.
BreakPoint Dialog allows user to modify properties.
IBreakpoint stores the information required to install a breakpoint in a specific type of debugger :
For example,
In case of a language debugger, a line breakpoint stores a fully qualified type name and line number.
In case of a graphical debugger, a figure breakpoint will store corresponding diagram model element uri and its location.
IDebugTarget Listens for breakpoints being added/removed/changed during its lifecycle, and updates them in the underlying runtime

Slide 87 of Debug_Platform_The_Basics

2. How to locate the code for a selected frame ?

Slides 91-105 of Debug_Platform_The_Basics

3. How to enable source highlighting for non-textual editor ?

Graphical Editors should implement IDebugEditorPresentation.
Debugger will send add/remove annotation events based on which editor will highlight the diagram elements.

4. How to display custom variable view an watch expression ?

slide 127 of Debug_Platform_The_Basics

Eclipse Debug UI Utilities interact with the Standard Debug Model.
Debug Model Elements – represent the program being debugged which has the capabilities to step, resume and terminate.
DebugPlugin.fireDebugEventSet(DebugEvent[] events)
DebugPlugin.addDebugEventListener(IDebugEventSetListener listener)

EventListner listens to the debug events and then accordingly refreshes the UI.




Reference : http://www.eclipsecon.org/2008/sub/attachments/Debug_Platform_The_Basics.ppt

No comments: