Network infrastructure application validation Engineers - Things to ask yourself

One of the challenges many Intelligent network infrastructure device vendors have is to ensure that their product withstands attacks on protocol processing logic of the device and also ensure that system withstands (robustness) even if the device is used beyond its marketed capabilities. Validation phase (Quality Assurance phase) is one of the phases in software engineering which must ensure  device robustness.  QA Engineers play one important role in product development.  Test Engineers, in my view, contribute to success of product more than development Engineers. Test Engineers need to be more agile, adaptive, understand product features, deployment scenarios and aware of all product features.  Only then  Test Engineers would be able to contribute more to validate the product. It is very easy for Test Engineers to get lost of above qualities and become dump black box testers running predefined functional tests again and again without adding value by creating new test cases with respect to attacks, scale and robustness.

Don't get me wrong.  Running the predefined tests on all releases is good to ensure that there is no regression.  But test engineers need to go beyond that be creative and think like attackers and think of multiple kinds of deployments. Creating test cases is continuous process.

I have written one blog entry on "advice to network equipment testers".  I hope that was useful.

What are the test Engineers need to know from their counter parts in development and product management. What questions you need to ask yourself as test Engineer.

1. Do I know the purpose of the feature which I am testing. 
2. What are deployment scenarios where this feature is applicable.
3. What are configuration parameters exposed by this feature to the end user.
4. How do I map these configuration parameters to the deployment scenarios.
5. Is this feature associated with any protocol processing?
6. If there is any protocol processing, do I know the protocol.
1. Go through the standard document if it standard protocol.
2. Go through your internal protocol documentation in case if it is proprietary protocol.
3. Go through the messages and fields AND possible values of protocols.
4. Find out if there are any tools in the public domain related to this feature.
   
5. Think like an attacker and note down the messages and fields for possible negative test cases.
   
7. What are the hardcoded parameters for this feature
1. How many configuration records can it take.
2. How many run time blocks can it create.
3. Think of deployments where the traffic can't be predicted.  These will help in creating test cases to test the robustness of the feature.
8. If there is any standard based protocol processing,  do I need to ensure that it is inter-operating with other products in the market.  
1. What are different ways this feature is used and interoperability testing should cover those.
9. What is the expected performance of this feature in low load and in high load conditions?
10. What is other traffic that is typically run when this feature is enabled. 
1. Find out all kinds of traffic that is possible in deployment scenarios. Ensure that this feature works along with other features.
11. Can the test cases automated?  
1. If so, automate them. So that next time you don't have to spend time on doing manual testing.
   

Above questions are not some thing new for you all. But if you don't know the answers for above questions, you are not doing good job for yourself and your organization.